indr 262 introduction to optimization methods linear algebra indr 262 metin türkay 1

INDR 262INTRODUCTION TO

OPTIMIZATION METHODS

LINEAR ALGEBRA

INDR 262 Metin Türkay 1

Matrices

• m and n are positive integers• Order of matrix: mxn• The number in the ith row and jth column of A is

called the ijth element of A and is written aij.

mnmm

n

n

aaa

aaa

aaa

A

..

........

..

..

21

22221

11211


Example

7

6

1

987

654

321

31

23

11

a

a

a

A


Equal Matrices

• Two matrices A and B are equal if and only if A and B are of the same order and for all i and j, aij=bij.

• If A=B, then x=1, y=2, w=3, z=4

1 2 B

3 4

x yA

w z


Vectors

• A list of n real numbers, say (a1, a2, …, an) is called an n-dimensional vector. An n-dimensional vector also maybe displayed as a 1 by n matrix.

321 ,2

1


Scalar Product of Two Vectors

• The scalar product of vectors

u = [u1 u2 … un] and

is u1v1+u2v2+…+unvn

nv

v

v

v..2

1


Example

• uv = 1x2+2x1+3x2 = 10

2

1

2

321 vu


Notes

• If u = [1 2 3] and , then uv is

not defined because the vectors are of different dimensions.

• Two vectors are perpendicular to each other if and only if their scalar product is equal to 0.

• E.g., u = [1 -1] and .

4

3v

1

1v


Scalar Multiple of a Matrix

• Given any matrix A and any scalar c, the scalar multiple of matrix A, cA, is obtained from the matrix A by multiplying each element of A by c.

03

633

01

21AA


Addition of two Matrices

• Let A=[aij] and B=[bij] be two matrices with the same order (say mxn). Then, the matrix C=A+B is defined to be the mxn matrix whose ijth element is aij+bij.

002

000

111120

332211

112

321

110

321

BAC

BA


The Transpose of a Matrix

• Given any mxn matrix

the transpose of A (written AT) is the nxm matrix

mnmm

n

n

aaa

aaa

aaa

A

..

........

..

..

21

22221

11211

nmnn

n

n

T

aaa

aaa

aaa

A

..

........

..

..

21

22212

12111


Example

• For any matrix A, (AT)T=A.

654

321

63

52

41

654

321 T TTAAA


Matrix Multiplication

• Given two matrices A and B, the matrix product of A and B is defined if and only if the number of columns in A is equal to the number of rows in B. The matrix product C=AB is determined as follows:

cij = scalar product of (row i of A and column j of B)


Properties of Matrix Multiplication

1. Matrix multiplication is associative, i.e., A(BC)=(AB)C.

2. Matrix multiplication is distributive, i.e., A(B+C)=AB+AC.


Example

53

1 12

42

1 12

43

1 11

32

1 11

22

21

12

11

c

c

c

c

ABC

BA

32

11

12

11

54

43ABC


Matrices and Systems of Linear Equations

• Consider a system of linear equations given by

• x1, x2, …, xn are referred to as variables• aij’s and bi’s are constants• A set of equations like above is called a linear system of m

equations in n variables

nnmnmm

nn

nn

bxaxaxa

bxaxaxa

bxaxaxa

...

...........................

...

...

2211

22222121

11212111


Solution

• A solution to a linear system of m equations in n unknowns is a set of values for the unknowns that satisfy each of the systems m equations.


Example

x1 + 2x2 = 5

2x1 - x2 = 0

2

1x

1

3x

00 0 ? 2)1(2

55 5 ? )2(21

05 0 ? 1)3(2

55 5 ? )1(23

Solution

Not a solution


Matrix Representation of Systems of Linear Equations

Ax = b

nnmnmm

n

n

b

b

b

b

x

x

x

x

aaa

aaa

aaa

A...

,...

,

...

............

...

...

2

1

2

1

21

22221

11211

mmnmm

n

n

b

b

b

aaa

aaa

aaa

bA...

...

............

...

...

2

1

21

22221

11211

Augmented matrix


The Gauss-Jordan Method for Solving Systems of Linear Equations

• Gauss-Jordan method is used to find solution(s) to systems of linear equations. A system of linear equations must satisfy one of the following cases:

• Case 1: The system has no solution• Case 2: The system has a unique solution• Case 3: The system has an infinite number of solutions


Elementary Row Operations• An ero transforms a given matrix A into a new matrix A’ via one

of the following operations.• Type 1 ero: A’ is obtained by multiplying any row of A by a

nonzero scalar.• Type 2 ero: Begin by multiplying any row of A (say, row i) by a

nonzero scalar c. For some ji, let row j of A’ = c(row i of A) + row j of A, and let the other rows of A’ be the same as the rows of A.

• Type 3 ero: Interchange any two rows of A.


Facts

• If the matrix A’ is obtained from A via an ero, A’ and A are equivalent.

• If the augmented matrix [A’b’] is obtained from [Ab] via an ero, the systems Ax=b and A’x=b’ are equivalent.

• Any sequence of ero’s performed on the augmented matrix [Ab] corresponding to the system Ax=b will yield an equivalent linear system.


Gauss-Jordan Method

• The Gauss-Jordan method solves a linear system of equations by utilizing ero’s in a systematic fashion.

Step 1 To solve Ax=b, write down the augmented matrix [Ab].

Step 2 At any stage, define a current row, current column, and a current entry. Begin with row 1 as the current row, column 1 as the current column, and a11 as the current entry.


Gauss-Jordan Method

a If a11 (the current entry) is nonzero, use ero’s to transform column 1 (the current column) to [1 0 … 0]T. Then, obtain the new current row, column, and entry by moving down one row and one column to the right, and go to Step 3.

b If a11 (the current entry) equals 0, then do a Type 3 ero involving the current row and any row that contains a nonzero entry in the current column. Use ero’s to transform column 1 (the current column) to [1 0 … 0]T. Then, obtain the new current row, column, and entry by moving down one row and one column to the right, and go to Step 3.

c If there are no nonzero numbers in the first column, obtain a new current column and entry by moving one column to the right. Then go to Step 3.


Gauss-Jordan Method

Step 3a If the new current entry is nonzero, use ero’s to transform it to 1 and the

rest of the current column’s entries to 0. When finished, obtain a new current row, column, and entry. If this is impossible, stop. Otherwise, repeat Step 3.

b If the current entry is 0, do a Type 3 ero with the current row and any row that contains a nonzero entry in the current column. Then, use ero’s to transform column entry to 1 and the rest of the current column’s entries to 0. When finished, obtain the new current row, column, and entry. If this is impossible, stop. Otherwise, repeat Step 3.

c If the current column has no nonzero numbers below the current row, obtain a new current column and entry and repeat Step 3. If it is impossible, stop.


Gauss-Jordan Method

Step 4 Write down the system of equations A’x=b’ that corresponds to the matrix [A’b’] obtained when Step 3 is completed. Then, A’x=b’ will have the same set of solutions as Ax=b.

• The Gauss–Jordan method converts the augmented matrix [Ab] into [A’b’] such that

'

'2

'1

...1...00

............

0...10

0...01

''

mb

b

b

bA''

22'11 ..., , , mn bxbxbx


Example

• Solve the following system of linear equations using the Gauss-Jordan method.

2x1 + 2x2 + x3 = 9

2x1 - x2 + 2x3 = 6

x1 - x2 + 2x3 = 5


Solution

5

6

9

211

212

122

bA

5

62

9

211

2122

111

11 bA

5

32

9

211

1302

111

22 bA

21

32

9

2320

1302

111

33 bA


Solution

2112

9

2320

31102

111

44 bA

2112

7

2320

31106

501

55 bA

2512

7

6500

31106

501

66 bA

3

12

7

1003

1106

501

77 bA


Solution

3

2

1

100

010

001

99 bA

3

1

1

1003

110

001

88 bA

x1 = 1 x2 = 2 x3 = 3


Analysis of the Solutions to Systems of Linear Equations

• For any linear system, a variable that appears with a coefficient of 1 in a single equation and a coefficient of 0 in all other equations is called a basic variable. Any variable that is not a basic variable is called a nonbasic variable.

x set of all of the variables in the system Ax=bxB set of all of the basic variables in the system Ax=bxN set of all of the nonbasic variables in the system

Ax=bx = xB xN


Analysis of the Solutions to Systems of Linear Equations

• The solution to A’x=b’ can be categorized in one of the three cases:

Case 1: A’x=b’ has at least one row of the form [0 0 … 0c] and c0. Then, Ax=b has no solution.

Case 2: When case 1 does not apply and xN=, then Ax=b has a unique solution.

Case 3: When case 1 does not apply and xN, then Ax=b has infinite number of solutions.


Example

• Case 1 does not apply since there are no rows of the form [0 0 … 0c] and c0.

• Case 2 does not apply since,

xB={x1, x2, x3}

xN={x4, x5}• There are infinite number of solutions.

0

1

2

3

00000

10100

02010

11001

'' bA


Example

• Assign arbitrary values to the variables in xN; x4=c, x5=k.• Write down the equations in [A’b’],

x1 + c + k = 3 x1 = 3 - c – kx2 + 2c = 2 x2 = 2 - 2cx3 + k = 1 x3 = 1 - k

• It is easy to see that there are infinite number of values of c and k that will satisfy this system of equations.

0

1

2

3

00000

10100

02010

11001

'' bA


Linear Combination

• A linear combination of the vectors in V is any vector of the form c1v1+c2v2+…+ckvk, where c1, c2, …, ck are arbitrary scalars.

• Example: V={[1,2], [2,1]}

2v1-v2 = 2([1 2]) – [2 1] = [0 3]

0v1-3v2 = 0([1 2]) – 3([2 1]) = [6 3]


Linear Independence & Linear Dependence

• A set V of m-dimensional vectors is linearly independent if the only linear combination of vectors in V that equals 0 is the trivial linear combination.

• A set V of m-dimensional vectors is linearly dependent if there is a nontrivial linear combination of the vectors in V that adds up to 0.


Example 1

• V={[1,0], [0,1]} Try to find a linear combination of vectors in V that yields 0.

• c1([1 0]) + c2([0 1]) = [0 0]

• In order to satisfy this, [c1 c2] = [0 0] c1=c2=0

• The only linear combination of vectors in V that yields 0 is the trivial linear combination. Therefore, V is a linearly independent set of vectors.


Example 2• V={[1,2], [2,4]} Try to find a linear combination of

vectors in V that yields 0.• c1([1 2]) + c2([2 4]) = [0 0]

[c1 2c1] + [2c2 4c2] = [0 0]c1 + 2c2 = 0 c1 = -2c2

2c1 + 4c2 = 0 2c1 = -4c2

• So, c1 = 2 c2 = -1 is one of the possible solutions.• There exists a nontrivial linear combination of vectors in

V that yields 0. Therefore, V is a linearly dependent set of vectors.


The Rank of a Matrix

• Let A be any mxn matrix, and denote the rows of A by r1, r2, …, rm. Also define R={ r1, r2, …, rm}.

• The rank of A is the number of vectors in the largest linearly independent subset of R.

• If for a matrix A with m rows, rank A=m; then the matrix is a collection of linearly independent set of vectors. If rank A<m; then the matrix contains a linearly dependent set of vectors.


Examples

3202

110

001

320

120

001

A

1002

110

001

100

010

001

Rank A = 3

011

010

001

rank B Rank B = 2


The Inverse of a Matrix• A single linear equation in a single variable can be solved

by multiplying both sides of the equation by multiplicative inverse of the variable coefficient.

• Example:4x=3 4-1(4x) = (4-1)3 x=3/4

• We can generalize this approach to square systems of linear equations (i.e., number of equations = number of unknowns).


Square and Identity Matrix

• A square matrix is any matrix that has an equal number of rows and columns.

• The diagonal elements of a square matrix are those elements aij such that i=j.

• A square matrix for which all diagonal elements are equal to 1 and all non-diagonal elements are equal to 0 is called an identity matrix.

100

010

001

, 10

0132 II


Inverse of A

• For a given mxm matrix A, the mxm matrix B is the inverse of A if

BA = AB = Im

100

010

001

201

715

101

101

213

102

AA-1=I


Finding the Inverse of a Matrix with the Gauss-Jordan Method

Step 1 Write down the mx2m matrix [AIm].

Step 2 Use ero’s to transform [AIm] into [ImB]. This will only be possible if rank A=m; in this case, B=A-1. If rank A<m, then A has no inverse.


Example

31

52A

10

01

31

522IA

10

021

312

5112IA

12

1

021

210

251

22IA

21

021

102

5132IA

21

53

10

0142IA

21

531A


Using Matrix Inverse to Solve Linear Systems of Equations

• Given a linear system of equations,

Ax=b• Multiply both sides by A-1

AA-1x = A-1b x = A-1b


Example

2x1 + 5x2 = 7

x1 + 3x2 = 4

4

7

31

52

2

1

x

x

21

53

31

52 1-AA

1

1

4

7

21

53

2

1

2

1

x

x

x

x


Determinant of a Matrix

• Any square matrix A has a number called the determinant of A (shown by set A or A).

122122112221

1211 det aaaaAaa

aaA


Minor of A

• If A is an mxm matrix; then for any values of i and j, the ijth minor of A (Aij) is the (m-1)x(m-1) submatrix of A obtained by deleting row i and column j of A.

12

11 ,

20

12

210

112

101

3212 AAA


Determinant of a Matrix

• Let A be any mxm matrix,

• det A = (-1)i+1ai1(det Ai1) + (-1)i+2ai2(det Ai2) + … +(-1)i+maim(det Aim)

mmmm

m

m

aaa

aaa

aaa

A

...

............

...

...

21

22221

11211


Example 1

210

112

101

A

1 1 1 2 1 31 1 2 1 2 1det ( 1) 1 det ( 1) 0 det ( 1) 1 det

1 2 0 2 0 1

det 1x1x 3 (-1)x0x4 1x1x2

det 5

A

A

A

det A = (-1)1+1 1 det A11 + (-1)1+2 0 det A12 + (-1)i+3 1 det Ai3


Example 2

5121

2122

1113

1201

A

det A = -48


indr 262 introduction to optimization methods linear algebra indr 262 metin türkay 1

Documents

metin trkay

x n matrix

b augmented matrix indr

example indr

matrix c

u n v n indr

n variables indr

solution indr