math5745 multivariate methods lecture 02sta6ajb/math5745/lecture02.pdf · math5745 multivariate...

45
MATH5745 Multivariate Methods Lecture 02 February 23, 2018 MATH5745 Multivariate Methods Lecture 02 February 23, 2018 1 / 45

Upload: hoangnga

Post on 23-May-2018

214 views

Category:

Documents


1 download

TRANSCRIPT

MATH5745 Multivariate MethodsLecture 02

February 23, 2018

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 1 / 45

Matrix operations

We are still discussing matrix operations.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 2 / 45

Matrix operationsMatrix multiplication in terms of vectors

We now consider matrix multiplication in terms of vectors.Let a, x1, x2, . . . , xn be all p × 1 and A be p × p.[QUIZ:] What is the size of each xi? How many xi ’s are there?We can write the following factoring results:

n∑i=1

a′xi = a′n∑

i=1

xi ,

n∑i=1

Axi = An∑

i=1

xi ,

n∑i=1

(a′xi )2 = a′

(n∑

i=1

xix′i

)a,

n∑i=1

Axi (Axi )′ = A

(n∑

i=1

xix′i

)A′.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 3 / 45

Matrix operationsMatrix multiplication in terms of vectors

How can we express matrix multiplication AB in terms of theirunderlying vectors?

Let a′i be the i-th row of A and b(j) be the j-th column of B.

The (i , j)-th element of AB is a′ib(j).

Example:

A =

a′1a′2a′3

, B =(b(1) b(2)

).

AB =

a′1b(1) a′1b(2)a′2b(1) a′2b(2)a′3b(1) a′3b(2)

.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 4 / 45

Matrix operationsMatrix multiplication in terms of vectors

The multiplication AB can be expressed in terms of the rows of A orthe columns of B.

Example:

A =

a′1a′2a′3

, B =(b(1) b(2)

).

AB =

a′1(b(1) b(2))a′2(b(1) b(2))a′3(b(1) b(2))

=

a′1Ba′2Ba′3B

=

a′1a′2a′3

B.

AB = A(b(1) b(2)) = (Ab(1) Ab(2)).

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 5 / 45

Matrix operationsMultiplications involving vectors

Any matrix can be multiplied by its transpose.

Suppose A is a n × p matrix.

Two most important matrix multiplications: A′A and AA′.

AA′ is n × n: product of rows of A.

A′A is p × p: product of columns of A.

[EXERCISE:] Show that A′A and AA′ are symmetric.

If both rows and columns of a matrix A are under discussion, we willuse the notation a′i for rows and a(j) for columns.

[QUIZ:] What are the range of i and j in A?

[QUIZ:] What are the sizes of a′i and a(j)?

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 6 / 45

Matrix operationsMatrix multiplication in terms of vectors

A can be written either as

A =

a′1a′2...a′n

or A =(a(1) a(2) . . . a(p)

)

Therefore A′A can be expressed as either row vectors

A′A = (a1 a2 . . . an)

a′1a′2...a′n

=n∑

i=1

aia′i

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 7 / 45

Matrix operationsMatrix multiplication in terms of vectors

. . . or column vectors

A′A =

a′(1)a′(2)

...a′(p)

(a(1) a(2) . . . a(p))

=

a′(1)a(1) a′(1)a(2) · · · a′(1)a(p)a′(2)a(1) a′(2)a(2) · · · a′(2)a(p)

......

. . ....

a′(p)a(1) a′(p)a(2) · · · a′(p)a(p)

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 8 / 45

Matrix operationsMatrix multiplication in terms of vectors

Similarly, AA′ can be expressed as row vectors

AA′ =

a′1a′2...a′n

(a1 a2 . . . an) =

a′1a1 a′1a2 · · · a′1ana′2a1 a′2a2 · · · a′2an

......

. . ....

a′na1 a′na2 · · · a′nan

. . . or column vectors

AA′ =(a(1) a(2) . . . a(p)

)

a′(1)a′(2)

...a′(p)

=

p∑j=1

a(j)a′(j)

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 9 / 45

Matrix operationsMatrix multiplication

Let A = (aij) be n × n and D = diag(d1, d2, . . . , dn) be a diagonalmatrix.

[EXERCISE:] What are the (i , j)-th element of:1 DA2 AD3 DAD4 IA5 AI6 IAI

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 10 / 45

Matrix operationsMatrix multiplication

Matrix multiplication of a matrix A of size n × p by a scalar c :

cA =

ca11 ca12 · · · ca1pca21 ca22 · · · ca2p

......

. . ....

can1 can2 · · · canp

.

Similarly, multiplication of a vector x of size n × 1 by a scalar c .

[EXERCISE:] What are the results of the multiplication of cI and cj?

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 11 / 45

Matrix operationsMatrix multiplication

The importance of the matrix multiplication with scalars is that itallows us to formulate linear combination of vectors or matrices.

Example:

k∑i=1

cixi = c1x1 + c2x2 + . . . + ckxk .

k∑i=1

ciBi = c1B1 + c2B2 + . . . + ckBk .

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 12 / 45

Matrix operationsMatrix multiplication

If A is a symmetric matrix and x and y are vectors of appropriate size,then

y′Ay =∑i

aiiy2i +

∑i 6=j

aijyiyj a quadratic form

x′Ay =∑ij

aijxiyj a bilinear form

[QUIZ:] What type of vector/matrix are they?

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 13 / 45

Partitioned matrices

A matrix can generally be partitioned into submatrices, for example

A =

2 1 3 8 4−3 4 0 2 7

9 3 6 5 −2

4 8 3 1 6

=

(A11 A12

A21 A22

).

If A and B are conformable and A and B are partitioned such thatthe submatrices are appropriately conformable, then

AB =

(A11 A12

A21 A22

)(B11 B12

B21 B22

)=

(A11B11 + A12B21 A11B12 + A12B22

A21B11 + A22B21 A21B12 + A22B22

).

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 14 / 45

Partitioned matrices

If A = (A1 A2), then A′ 6= (A′1 A′2) but

A′ =

(A′1A′2

).

Example:

A =

(2 1 3 8 4−3 4 0 2 7

)=(A1 A2

).

A′ =

2 −31 43 0

8 24 7

=

(A′1A′2

).

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 15 / 45

Matrix rankLinear dependency

Let us consider first the concept of linear dependency.

A set of vectors a1, a2, . . . , an are linearly dependent if

c1a1 + c2a2 + . . . + cnan = 0

for some constants c1, c2, . . . , cn (not all zero).

If no such constants can be found, then the set of vectors are linearlyindependent.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 16 / 45

Matrix rank

Example: Suppose

a1 =

124

, a2 =

−251

, a3 =

099

.

Can see that a1, a2 and a3 are linearly dependent because whenc1 = 2, c2 = 1, c3 = −1, then

2

124

+

−251

− 0

99

=

000

.

If you remove any one vector from the set, you will have linearlyindependent vectors.

At the most, you can have two linearly-independent vectors from theset.

If those vectors are in a matrix, we say that the matrix is of rank 2.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 17 / 45

Matrix rank: Example

Recall the above comment:“If those vectors are in a matrix, we say that the matrix is of rank 2.”

Example: Suppose

A =

1 −2 02 5 94 1 9

.

Here A has rank 2.

Notice that 2a(1) + a(2) − a(3) = 0.Hence A has two linearly independent columns.

Notice that 2a′1 + a′2 − a′3 = 0.Hence A has two linearly independent rows.

This is an example of the result:The rank of any square or rectangular matrix A is defined as thenumber of linearly independent rows or columns of A.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 18 / 45

Matrix rank

The rank of any square or rectangular matrix A is defined as thenumber of linearly independent rows or columns of A.

If A is a n × p matrix the maximum possible rank of A is the smallerof n and p.

In such case, we say that A is of full rank.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 19 / 45

Matrix rank example

Example: Here A =

1 5−2 2

3 4

is of full rank and has rank 2.

The columns of A are linearly independent. [How do I know?]

The columns are linearly independent so there are NO values c1 andc2 such that

c1

1−5

3

+ c2

524

=

000

.

This can be written as 1 5−2 2

3 4

( c1c2

)=

000

or Ac = 0.

The only solution is c = (0, 0)′.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 20 / 45

Matrix rank example

Example: Here A is of full rank and has rank 2.

A =

(1 −2 35 2 4

).

The rows of A are linearly independent. [How do I know?]The columns are linearly dependent as we can find c1, c2, c3 such that

c1

(15

)+ c2

(−2

2

)+ c3

(34

)=

(00

).

This can be written as(1 −2 35 2 4

) c1c2c3

=

(00

)or

Ac = 0.

A solution is given by multiples of c = (14 , −11 , −12)′.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 21 / 45

Matrix rank

In the previous example we had a solution c 6= 0 of the equation

Ac = 0.

Consequence of linear dependencies among columns of A: the productof A and c is equal to 0 even though A 6= O and c 6= 0.

It is possible to have the equation AB = CB, and yet A 6= C forrectangular matrices A,B, and C.

Therefore we cannot, in general, cancel matrices from both sides of anequation.

Exceptions:1. Nonsingular matrix (later).2. The equation holds for all possible values of the matrix common inboth sides of the equation.Thus if Ax = Bx for all possible values of x, then A = B.

(A simple example)

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 22 / 45

Matrix rank: Example

It is possible to have the equation AB = CB, and yet A 6= C forrectangular matrices A,B, and C.

Example:

Let A =

(2 31 3

), C =

(1 10 1

), B =

(2 2−1 −1

).

Then

AB =

(1 1−1 −1

).

Also

CB =

(1 1−1 −1

).

Thus AB = CB, but A 6= C.Thus (A− C)B = O but (A− C) 6= O and B 6= O.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 23 / 45

Matrix inverse

If a matrix A is square and of full rank, then A is nonsingular and hasa unique inverse denoted A−1 such that

AA−1 = A−1A = I.

For example,

A =

(3 42 6

)A−1 =

(0.6 −0.4−0.2 0.3

).

AA−1 =

(3 42 6

)(0.6 −0.4−0.2 0.3

)=

(1 00 1

).

If A is square and less than full rank, then the inverse does not existand A is said to be singular.

Rectangular matrices do not have an inverse (with the abovedefinition).

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 24 / 45

Matrix inverse

If A and B are of the same size and are nonsingular, then

(AB)−1 = B−1A−1.

Note: (A′A)−1 cannot be expressed as A−1(A′)−1 if A is rectangular.

If a matrix is nonsingular, it can be cancelled from both sides of anequation (because they have inverses).

Example: If B is nonsingular, then

AB = CB implies A = C.

Multiply both sides of the equation with B−1 on the right side

ABB−1 = CBB−1

AI = CI

A = C

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 25 / 45

Matrix inverse

For a nonsingular matrix

(A′)−1 = (A−1)′.

Example:

A =

(3 72 5

), A−1 =

(5 −7−2 3

).

A′ =

(3 27 5

), (A′)−1 =

(5 −2−7 3

).

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 26 / 45

Positive definite matrices

Recall that a quadratic form satisfies

x′Ax =∑i

aiix2i +

∑i 6=j

aijxixj .

A symmetric matrix A is positive definite if x′Ax > 0 for all possiblex 6= 0.

It is positive semi-definite if x′Ax ≥ 0 for all possible x 6= 0.

The diagonal elements aii are positive for all i .

If for example a11 < 0, then x′ = (1, 0, 0, . . . , 0) would give x′Ax < 0.Recall that to be positive definite x′Ax > 0 for ALL possible x 6= 0.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 27 / 45

Positive definite matrices

If A = B′B, where B is n × p of full rank p (with p < n), then B′B ispositive definite.

We can write

x′Ax = x′B′Bx = (Bx)′(Bx) = z′z

where z = Bx Therefore x′Ax =n∑

i=1

z2i which is positive.

Notice: Bx cannot be 0 unless x = 0 because B is full rank

Note the result is analoguous to a = b2 in real numbers.

Positive definite matrices can be factored out into ‘square root’ in twoways (very useful later in our discussion).

First, via spectral decomposition (later).Second, via Cholesky decomposition.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 28 / 45

Positive definite matricesCholesky decomposition

A positive definite matrix A can be written as A = T′Twhere T is a nonsingular upper diagonal matrix.

See the textbook or other sources about this.

We’ll discuss the interpretation later.

Some textbooks define Cholesky1 decomposition using lower triangularmatrices.

1Andre-Louis Cholesky (1875-1918) was a French military officer.MATH5745 Multivariate Methods Lecture 02 February 23, 2018 29 / 45

Positive definite matricesCholesky decomposition

A positive definite matrix A can be written as

A = T′T

where T is a nonsingular upper diagonal matrix.Example:

If A =

3 0 −30 6 3−3 3 6

then T =

3 0 −√

3

0√

6√

3/2

0 0√

3/2

.

T′T =

3 0 0

0√

6 0

−√

3√

3/2√

3/2

3 0 −√

3

0√

6√

3/2

0 0√

3/2

=

3 0 −30 6 3−3 3 6

= A.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 30 / 45

Matrix determinants

Consider a 2× 2 matrix A (square matrix only!)

A =

(a11 a12a21 a22

).

The determinant of A, denoted |A|, is given by (a scalar!)

|A| = a11a22 − a21a12.

Larger matrices follow the principle for a 3× 3 matrix:∣∣∣∣∣∣a b cd e fg h i

∣∣∣∣∣∣ = a

∣∣∣∣ e fh i

∣∣∣∣− b

∣∣∣∣ d fg i

∣∣∣∣+ c

∣∣∣∣ d eg h

∣∣∣∣and the 2× 2 determinant follows the above formula.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 31 / 45

Matrix determinantsProperties

Switching two rows or columns changes the sign.

Example: ∣∣∣∣ 2 25 6

∣∣∣∣ = 2(6)− 2(5) = 2.∣∣∣∣ 5 62 2

∣∣∣∣ = 5(2)− 6(2) = −2.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 32 / 45

Matrix determinantsProperties

Multiplying one row by a constant multiplies the whole determinantby that constant.

Example: ∣∣∣∣ 4 45 6

∣∣∣∣ = 4(6)− 4(5) = 4.

It follows that if A is of size n × n and full rank, then |cA| = cn|A|.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 33 / 45

Matrix determinantsProperties

For conformable square matrices A and B, |AB| = |A||B|.Example:

Let A =

(2 11 2

)with |A| = 3.

Let B =

(3 24 1

)with |B| = −5.

Then AB =

(10 511 4

)with |AB| = −15.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 34 / 45

Matrix determinants

If matrix A is of reduced rank, then there is a linear dependencybetween columns or rows and in this case |A| = 0.

We can say that when |A| = 0 then A is of reduced-rank and singular,and hence it does not have an inverse.

Example: ∣∣∣∣ 2 34 6

∣∣∣∣ = 2(6)− 3(4) = 0.

Example:∣∣∣∣∣∣1 −2 02 5 94 1 9

∣∣∣∣∣∣ = (1)

∣∣∣∣ 5 91 9

∣∣∣∣− (−2)

∣∣∣∣ 2 94 9

∣∣∣∣+ (0)

∣∣∣∣ 2 54 1

∣∣∣∣= (1)(36)− (−2)(−18) + (0)(−18) = 0.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 35 / 45

Matrix determinants

If D = diag(d1, d2, . . . , dn), then

|D| =n∏

i=1

di .

Similarly |In| = 1 and |cIn| = cn.

If A is positive definite, |A| > 0.

|A′| = |A| and |A−1| = 1/|A| = |A|−1.

Interpretation: The determinant of the linear transformation (matrix)A is the (signed) volume of the region obtained by applying A to theunit cube.

(Some examples)

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 36 / 45

Matrix determinants: Example 1

Let A =

(2 11 2

)with |A| = 3.

This maps the points (x , y) = (0, 0), (0, 1), (1, 0), (1, 1) as follows:

A

(00

)=

(00

), A

(01

)=

(12

), A

(10

)=

(21

), A

(11

)=

(33

).

The transformation A maps unit square to a shape with area 3.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 37 / 45

Matrix determinants: Example 1

The transformation A maps unit square to a shape with area 3.

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 38 / 45

Matrix determinants: Example 2

Let A =

(2 00 2

)with |A| = 4.

This maps the points (x , y) = (0, 0), (0, 1), (1, 0), (1, 1) as follows:

A

(00

)=

(00

), A

(01

)=

(02

), A

(10

)=

(20

), A

(11

)=

(22

).

The transformation A maps unit square to a square with area 4.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 39 / 45

Matrix determinants: Example 2

The transformation A maps unit square to a square with area 4.

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 40 / 45

Matrices as transformations

A matrix A can be interpreted as giving a transformation.

Suppose x = Ax.

Then x can be thought of as a transformation of x by the matrix A.

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 41 / 45

Matrices as transformations: Example 1

Suppose A =

(2 11 1

)so |A| = 1.

Then A

(10

)=

(21

), A

(01

)=

(11

), A

(11

)=

(32

).

−3 −2 −1 0 1 2 3

−3

−2

−1

0

1

2

3

−3 −2 −1 0 1 2 3

−3

−2

−1

0

1

2

3

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 42 / 45

Matrices as transformations: Example 2

Suppose A =

(2 −11 1

)so |A| = 3.

Then A

(10

)=

(21

), A

(01

)=

(−1

1

), A

(11

)=

(12

).

−3 −2 −1 0 1 2 3

−3

−2

−1

0

1

2

3

−3 −2 −1 0 1 2 3

−3

−2

−1

0

1

2

3

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 43 / 45

Matrices as transformations: Example 3

Suppose A =

(2 −11 −1

)so |A| = −1.

Then A

(10

)=

(21

), A

(01

)=

(−1−1

), A

(11

)=

(10

).

−3 −2 −1 0 1 2 3

−3

−2

−1

0

1

2

3

−3 −2 −1 0 1 2 3

−3

−2

−1

0

1

2

3

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 44 / 45

Matrices as transformations: Example 4

Suppose A =

(2 −1−1 −1

)so |A| = −3.

Then A

(10

)=

(2−1

), A

(01

)=

(−1−1

), A

(11

)=

(1−2

).

−3 −2 −1 0 1 2 3

−3

−2

−1

0

1

2

3 ●

●−3 −2 −1 0 1 2 3

−3

−2

−1

0

1

2

3

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●●

●●●

MATH5745 Multivariate Methods Lecture 02 February 23, 2018 45 / 45