MIDDLE EAST TECHNICAL UNIVERSITY NORTHERN CYPRUS CAMPUS

MATH 210 SPRING 2015

Week 9: Projections and Data Fitting

All linear combinations x1v1 + x2v2 + . . .+ xnvn are called the linear span of the vectors v1,v2, · · · ,vn . If A = [v1v2 . . .vn]is the matrix whose columns consists of the vectors, v1,v2, · · · ,vn , then A ·x = x1v1 + x2v2 + . . .+ xnvn . The linear span ofv1,v2, · · · ,vn are all the vectors of the form A ·x.

If for a given vector b, one can solve Ax = b for some vector x, then b is a linear combination of v1,v2, · · · ,vn . Moreprecisely, b = x1v1 +x2v2 + . . .+xnvn .

What if you cannot solve Ax = b for x? What is the next best solution? One can try the minimize the length of Ax−b.Clearly, if the minimum length is zero, then Ax−b = 0 is solved. Typically, minimization problems related to lengths/distances,it is better to deal with the square of the lengths/distances.

We denote the length of a vector w by ||w||. In fact, ||w||2 = wT w. In this sense, we would like to minimize ||Ax−b||2 =(Ax−b)T (Ax−b).

Intuition tells us that the vector b must split as a sum, b = p+e, where p = A · x̂ for some x̂ and e is a vector perpendicularto all A ·x’s.

This leads us to the following formulae/equations:

(AT A) · x̂ = AT ·b, (The “Normal Equation”) (0.1)

p = A · x̂ = A(AT A)−1 AT ·b, (0.2)

e = b−p = b − A(AT A)−1 AT ·b. (0.3)

Typically, the vector p is called the projection of the vector b onto the plane determined by the vectors v1,v2, · · · ,vn . Thevector x̂ is the coefficients needed to express the vector p as a linear combination of the v1,v2, · · · ,vn . The vector e isorthogonal component of the vector b which is perpendicular to the plane determined by the vectors v1,v2, · · · ,vn .

1 PROJECTION.

Project the vector b on the plane determined by the vectors v1,v2, · · · ,vn : Calculate the vectors p, e and x̂. Compute thesquared error ||e||2 = eT e.

(a) v1 =[2]

, b = [6]

.

(b) v1 =[0]

, b = [5]

.

(c) v1 =[

32

], b =

[58

].

(d) v1 =[

32

], v2 =

[26

], b =

[58

].

(e) v1 =[

11

], v2 =

[22

], b =

[53

].

(f) v1 =[

11

], v2 =

[22

], b =

[55

].

(g) v1 =1

01

, v2 =1

10

, b =1

11

.

(h) v1 =

1011

, v2 =

1100

, b =

1111

.

2 APPROXIMATE SOLUTIONS OF A MATRIX EQUATION.

For the equations Ax = b below, write the “normal equation” and find the best x̂ so that ||b−Ax̂|| is minimized.

(a)[4]

x = [8].

(b)[7]

x = [0].

(c)

[23

]x =

[87

].

(d)

[7 −2

−2 4

]x =

[58

].

(e)

[4 66 9

]x =

[53

].

(f )

[4 66 9

]x =

[23

].

(g)

1 01 −10 1

x =1

11

.

(h)

1 11 21 31 4

x =

13

1014

.

HINT: Remember that AT b =

v1 •bv2 •b

...

is the vector of dot products of b with the columns of A

and similarly AT A =

v1 •v1 v1 •v2 · · ·v2 •v1 v2 •v2 · · ·

......

. . .

is the matrix of dot products of columns of A with each other.

3 DATA FITTING ( THE LEAST SQUARES METHOD).

In this problem, you will be given a data set and the a function with some unknown coefficients. The aim is a) to formu-late this problem to find the best coefficients for the function and b) find the best coefficients.

a) For the following data sets and the function f (t ), formulate how you can find the best coefficients for the functionf (t ). Setup your problem in the form Ax = b so that

• Each row corresponds to a data point,

• The vector x is the vector of unknown coefficients in f (t ),

• The vector b consists of the ”observed values” of f (t ).

Do not solve for x.

1) f (t ) = a cos(t )+b sin(t )+ c.

t 0 π6

π3

π2

f (t ) 1 -1 3 2

2) f (t ) = a cos(t )+b sin(t )+ ce t +d ln(t ).

t π6

π3

π2

3π4 π 3π

2f (t ) -1 3 2 0 5 -7

b) Same as a) above, but also solve for x.

1) f (t ) = at +bt 3.

t -1 0 1f (t ) -3 1 4

2) f (t ) = a +bt 2.

t -1 0 1f (t ) 3 -1 4

3) f (t ) = a.

t -1 0 1f (t ) 3.1 2.9 3

4) f (t ) = a 1t +b.

t 13

12 1

f (t ) 6 0 -6

5) f (t ) = a cos(t )+b sin(t ).

t 0 π6

π3

π2

f (t ) 1 -1 3 2