chapter 6 orthogonality and least squares -...
TRANSCRIPT
![Page 1: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/1.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 1
Chapter 6 Orthogonality and Least Squares
6.3 ORTHOGONAL PROJECTIONS
![Page 2: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/2.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 2
ORTHOGONAL PROJECTIONS
The orthogonal projection of a point in onto a line
through the origin has an important analogue in .
Given a vector y and a subspace W in , there is a
vector in such that (1) is the unique vector in W
for which is orthogonal to W, and (2) is the
unique vector in W closest to y. See Fig. 1. These two
properties of provide the key to finding the
least-squares solutions of linear systems(Section 6.5).
![Page 3: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/3.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 3
Figure 1
Example 1: Let … be an orthogonal basis for
and let
![Page 4: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/4.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 4
⋯
Consider the subspace W=Span , and write as the
sum of a vector in and a vector in ⊥.
Solution : Write
where is in Span and
is in Span . To show that
is in ⊥, it suffices to show that is orthogonal to
![Page 5: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/5.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 5
the vectors in the basis for . Using properties
of the inner product, compute
∙ ∙ ∙ ∙ ∙
because is orthogonal to , , and . A similar
calculation shows that ∙ . Thus is in ⊥. ■
Theorem 8: The Orthogonal Decomposition Theorem
Let W be a subspace of . Then each y in can be
![Page 6: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/6.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 6
written uniquely in the form
----(1)
where is in W and is in ⊥. In fact, if … is any orthogonal basis of W, then
∙
∙ ⋯∙
∙ ----(2)
and .
The vector in (1) is called the orthogonal projection of
onto W and often is written as projW y. See Fig. 2.
![Page 7: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/7.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 7
Proof : Let … be any orthogonal basis for W,
and define by (2). Then is in W because is a
linear combination of the basis … . Let .
![Page 8: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/8.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 8
Since is orthogonal to … it follows from (2)
that
∙ ∙ ∙ ∙∙ ∙ ⋯
∙ ∙
Thus z is orthogonal to . Similarly, z is orthogonal to
each in the basis for W. Hence z is orthogonal to
every vector in W. That is, z is in ⊥.
To show that the decomposition in (1) is unique,
suppose y can also be written as with in W
![Page 9: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/9.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 9
and in ⊥. Then (since both sides equal
y), and so
This equality shows that the vector is in W and
in ⊥(because and z are both in ⊥, and ⊥ is a
subspace). Hence ∙ , which shows that . This
proves that and . ■
The uniqueness of the decomposition (1) shows that
![Page 10: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/10.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 10
the orthogonal projection depends only on W and not
on the particular basis used in (2).
Example 1: Let
and
.
Observe that is an orthogonal basis for
Spanu u. Write as the sum of a vector in and a vector orthogonal to .
Solution : The orthogonal projection of onto is
![Page 11: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/11.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 11
∙
∙ ∙
∙
Also
Theorem 8. ensures that is in ⊥. To check the
calculations, however, it is a good idea to verify that
![Page 12: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/12.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 12
is orthogonal to both and and hence to all of
. The desired decomposition of is
A Geometric Interpretation of the Orthogonal Projection
See Figure 3 and Figure 4 in Section 6.2.
![Page 13: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/13.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 13
![Page 14: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/14.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 14
Properties of Orthogonal Projections If … is an orthogonal basis for W and if y
happens to be in W, then the formula for projWy is
exactly the same as the representation of y given in
Theorem 5 in Section 6.2. In this case, projW y y.
If y is in = Span … , then projW y y.
Theorem 9 THE BEST APPROXIMATION THEOREM
Let W be a subspace of , let y be any vector in ,
![Page 15: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/15.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 15
and let be the orthogonal projection of y onto W. Then
is the closest point in W to y, in the sense that
∥∥∥∥ ----(3)
for all v in W distinct from .
The vector in Theorem 9 is called the best
approximation to y by elements of W. The distance from
y to v, given by ∥∥, can be regarded as the
"error" of using v in place of y. Theorem 9 says that
![Page 16: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/16.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 16
this error is minimized when .
Inequality (3) leads to a new proof that does not
depend on the particular orthogonal basis used to
compute it. If a different orthogonal basis for W were
used to construct an orthogonal projection of y, then this
projection would also be the closest point in W to y,
namely, .
Proof : Take v in W distinct from . See Fig. 4. Then
is in W. By the Orthogonal Decomposition Theorem,
![Page 17: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/17.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 17
is orthogonal to W.
![Page 18: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/18.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 18
In particular, is orthogonal to (which is in W ).
Since
the Pythagorean Theorem gives
∥∥ ∥∥∥∥
(See the colored right triangle in Fig. 4. The length of
each side is labeled.) Now ∥∥ because
≠, and so inequality (3) follows immediately.
![Page 19: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/19.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 19
Example 3: If
, and
Spanu u, as in Example 2, then the closest point
in to y is
∙
∙ ∙
∙
Example 4: The distance from a point y in to a
subspace W is defined as the distance from y to the
![Page 20: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/20.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 20
nearest point in W. Find the distance from y to
Spanu u, where
1 2
1 5 1y 5 ,u 2 ,u 2
10 1 1
Solution : By the Best Approximation Theorem, the
distance from y to W is ∥∥, where projW y.
Since {u1, u2} is an orthogonal basis for W,
![Page 21: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/21.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 21
1 2
2 2 2
5 1 115 21 1 7y u u 2 2 830 6 2 2
1 1 4
1 1 0ˆy y 5 8 3
10 4 6
ˆy y 3 6 45
The distance from y to W is .
![Page 22: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/22.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 22
Theorem 10: If {u1, ,up} is an orthonormal basis for a
subspace W of , then
projW ∙ ∙ ⋯∙ ----(4)
If ⋯ , then
projW for all y in ----(5)
Proof : Formula (4) follows immediately from (2) in
Theorem 8. Also, (4) shows that projWy is a linear
combination of the columns of U using the weights
∙ … ∙ . The weights can be written as
![Page 23: Chapter 6 Orthogonality and Least Squares - WKUrg.wonkwang.ac.kr/teaching/2.Linear-Algebra/Linear... · 2020-02-25 · Linear Algebra-chapter06-03-2013.hwp Author: Chae Created Date:](https://reader030.vdocuments.mx/reader030/viewer/2022040413/5f10cb997e708231d44ada85/html5/thumbnails/23.jpg)
선형대수학 6.3 2013
원광대 수학과 채갑병 23
… , showing that they are the entries in UTy
and justifying (5).