MTH 309 Y LECTURE 18.1 TUE 2014.04.08

Recall:1) A least square solution of a matrix equation Ax = b is a solution

of the equation

Ax = proj

Col(A)

b

3) If {w1

� � � � � w�} is an orthogonal basis of a subspace V ⊆ R�then for u ∈ R�

we have

projV u =

� u · w1

w1

· w1

�

w1

+

� u · w2

w2

· w2

�

w2

+ · · · +

� u · w�w� · w�

�

w�

4) If {v1

� � � � � v�} is any basis of a subspace V ⊆ R�then using

the Gram-Schmidt process we can compute an orthogonal basis

of V .

2) If Ax = b is a consistent equation then b ∈ Col(A), and so

proj

Col(A)

b = b. In such case we have:

�

least square solutions

of Ax = b

�

=

�

the usual solutions

of Ax = b

�

If Ax = b is inconsistent then least square solutions are the

best possible substitute for (nonexistent) solutions of this equa-

tion.

MTH 309 Y LECTURE 18.2 TUE 2014.04.08

How to compute least square solutions of Ax = b(version 1.0)¿ Calculate a basis for Col(A).

¡ Use the Gram-Schmidt process to compute an orthogonal

basis of Col(A).

¬ Use the orthogonal basis to compute proj

Col(A)

b.

√ Compute solutions of the equation Ax = proj

Col(A)

b.

Next goal: Simplify this.

MTH 309 Y LECTURE 18.3 TUE 2014.04.08

Definition. If V is the a subspace of R�

then the orthogonal

complement of V is the set V

⊥

of all vectors orthogonal to V :

V

⊥

= {w ∈ R�

| w · v = 0 for all v ∈ V }

V ⊥

V

Proposition. If V is a subspace of R�then V ⊥

is also a subspace

of R�.

MTH 309 Y LECTURE 18.4 TUE 2014.04.08

Recall: If A is an � × � matrix then Row(A) is the subspace of

R�spanned by the rows of A.

Proposition. If A is an � × � matrix then

Row(A)

⊥= Nul(A)

MTH 309 Y LECTURE 18.5 TUE 2014.04.08

Corollary. If A is an � × � matrix then

Col(A)

⊥= Nul(AT

)

MTH 309 Y LECTURE 18.6 TUE 2014.04.08

Definition. The equation

(A

T

A)x = A

T

b

is called the normal equation of Ax = b.

Back to least square solutions

Theorem. A vector x̂ is a least square solution of the equation

Ax = b

iff x̂ is an ordinary solution of the equation

(AT A)x = ATb

Example. Compute least square solutions of the following equation:

⎡

⎣

1 1

0 2

0 0

⎤

⎦ ·� �

1

�2

�

=

⎡

⎣

1

2

3

⎤

⎦

MTH 309 Y LECTURE 18.7 TUE 2014.04.08

How to compute least square solutions of Ax = b(version 2.0)¿ Compute AT A, AT

b.

¡ Solve the normal equation (AT A)x = ATb.

MTH 309 Y LECTURE 18.8 TUE 2014.04.08

Application: least square lines

Definition. If (�1

� �1

)� � � � � (��� ��) are points on the plane then

the least square line for these points is the line given by the

equation

� (�) = �� + �such that the number

dist

⎛

⎝

⎡

⎣

�1

.

.

.

��

⎤

⎦ �⎡

⎣

� (�1

)

.

.

.

� (��)

⎤

⎦

⎞

⎠

=

�

(�1

− � (�1

))

2

+ � � � (�� − � (��))

2

is as small as possible.

� (�) = �� + �

�1 �2 �3 �4

�4

� (�4)� (�3)

�3

�2

�1

� (�1)� (�2)

MTH 309 Y LECTURE 18.9 TUE 2014.04.08

Proposition. The line � (�) = �� + � is the least square line for

the points (�1

� �1

)� � � � � (��� ��) if the vector

���

�

is a least square

solution of the equation

⎡

⎣

�1

1

.

.

.

.

.

.

�� 1

⎤

⎦ ·� �

1

�2

�

=

⎡

⎣

�1

.

.

.

��

⎤

⎦

Upshot:

MTH 309 Y LECTURE 18.10 TUE 2014.04.08

Example. Find the equation of the least square line for the points

(0� 0)� (1� 1)� (3� 1)� (5� 3).

Definition. If (�

1

� �

1

)� � � � � (�

�

� �

�

) are points on the plane then

the least square parabola for these points is the parabola given

by the equation

� (�) = ��

2

+ �� + �

such that the number

dist

⎛

⎝

⎡

⎣

�

1

.

.

.

�

�

⎤

⎦

�

⎡

⎣

� (�

1

)

.

.

.

� (�

�

)

⎤

⎦

⎞

⎠

=

�

(�

1

− � (�

1

))

2

+ � � � (�

�

− � (�

�

))

2

is as small as possible.

MTH 309 Y LECTURE 18.11 TUE 2014.04.08

Least square curvesThe above procedure can be used to determine curves other than

lines that fit a set of points in the least square sense.

Example: least square parabolas.

(�1� � (�1))

(�1� �1)

� (�)= ��2 + �� + �

MTH 309 Y LECTURE 18.12 TUE 2014.04.08

Proposition. The parabola � (�) = ��2

+�� +� is the least square

parabola for the points (�1

� �1

)� � � � � (��� ��) if the vector

⎡

⎣

���

⎤

⎦

is

a least square solution of the equation

⎡

⎣

�2

1

�1

1

.

.

.

.

.

.

.

.

.

�2� �� 1

⎤

⎦ ·⎡

⎣

�1

�2

�3

⎤

⎦

=

⎡

⎣

�1

.

.

.

��

⎤

⎦

Upshot:

MTH 309 Y LECTURE 18.13 TUE 2014.04.08

Example. Find the least square parabola � (�) = ��2

+�� +� for the

following set of points:

(−2� 2)� (0� 0)� (1� 1)� (2� 3)