lecture slides linalg

7/29/2019 Lecture Slides Linalg

1/23

Financial Engineering and Risk ManagementReview of vectors

Martin Haugh Garud Iyengar

Columbia UniversityIndustrial Engineering and Operations Research


2/23

Reals numbers and vectors

We will denote the set of real numbers by R

Vectors are finite collections of real numbers

Vectors come in two varieties

Row vectors: v =v1 v2 . . . vn

Column vectors w =

w1w2

.

..wn

By default, vectors are column vectors

The set of all vectors with n components is denoted by Rn

2


3/23

Linear independence

A vector w is linearly dependent on v1, v2 if

w =1

v1

+2

v2

for some1, 2

R

Example:

2

6

4

= 2

1

1

0

+ 4

0

1

1

Other names: linear combination, linear spanA set V= {v1, . . . , vm} are linearly independent if no vi is linearlydependent on the others, {vj : j= i}

3


4/23

Basis

Every w Rn is a linear combination of the linearly independent set

B =

1

0

...0

,

0

1

...0

, . . .

0

0

...1

w = w1

1

0

...0

e1+w2

0

1

...0

e2+ . . . + wn

0

0

...1

enBasis any linearly independent set that spans the entire spaceAny basis for Rn has exactly n elements

4


5/23

Norms

A function (v) of a vector v is called a norm if

(v) 0 and (v) = 0 implies v = 0

(v) = || (v) for all R

(v1 + v2) (v1) + (v2) (triangle inequality)

generalizes the notion of length

Examples:2 norm: x2 =

ni=1

|xi|2 ... usual length

1 norm: x1 =n

i=1|xi|

norm: x = max1in |x|i

p norm, 1 p < : xp =n

i=1|x|pi

1p

5


6/23

Inner product

The inner-product or dot-product of two vector v, w Rn is defined as

v w =n

i=1

viwi

The 2 norm v2 =

v vThe angle between two vectors v and w is given by

cos() =v w

v2w

2

Will show later: v w = vw = product of v transpose and w

6


7/23

Financial Engineering and Risk ManagementReview of matrices




8/23


9/23

Matrix Operations: Transpose

Transpose: A Rmd

A =

a11 a12 . . . a1da21 a22 . . . a2d

......

. . ....

am1 am2 . . . amd

=

a11 a21 . . . am1a12 a22 . . . ad2

......

. . ....

a1d a2d . . . amd

Rdm

Transpose of a row vector is a column vector

Example:

A =

2 3 7

1 6 5

: 2 3 matrix ... A =

2 1

3 6

7 5

: 3 2 matrix

v =

26

4

: column vector ... v =

2 6 4

: row vector

3


10/23

Matrix Operations: Multiplication

Multiplication: A Rmd, B Rdp then C = AB Rmp

cij =ai1 ai2 . . . aid

b1j

b2j...bdj

row vector v R1d times column vector w Rd1 is a scalar.

Identity times any matrix AIn = ImA = A

Examples:

2 3 7

1 6 52

6

4 = 2(2) + 3(6) + 7(4)1(2) + 6(6) + 5(4) =

50

582 norm:

1

2

2

=

12 + (2)2 =

1 2

12

=

1

2

1

2

inner product: v w = vw

4


11/23

Linear functions

A function f : Rd Rm is linear if

f(x + y) = f(x) + f(y), , R, x, y

Rd

A function f is linear if and only if f(x) = Ax for matrix A Rmd

Examples

f(x) : R3 R: f(x) = 2 3 4x1x2x3

= 2x1 + 3x2 + 4x3f(x) : R3 R2: f(x) =

2 3 4

1 0 2

x1x2x3

=

2x1 + 3x2 + 4x3

x1 + 2x3

Linear constraints define sets of vectors that satisfy linear relationships

Linear equality: {x : Ax = b} ... line, plane, etc.

Linear inequality: {x : Ax b} ... half-space

5


12/23

Rank of a matrix

column rank of A Rmd = number of linearly independent columnsrange(A) = {y : y = Ax for some x}

column rank of A = size of basis for range(A)

column rank of A = m range(A) = Rm

row rank of A = number of linearly independent rows

Fact: row rank = column rank min{m, d}Example:

A =

1 2 3

2 4 6

, rank = 1, range(A) =

1

2

: R

A Rnn and rank(A) = n A invertible, i.e. A1 Rnn

A1A = AA1 = I

6


13/23

Financial Engineering and Risk ManagementReview of linear optimization




14/23

Hedging problem

d assets

Prices at time t= 0: p Rd

Market in m possible states at time t= 1

Price of asset j in state i = Sij

Sj =S1j

S2j...Smj

S = S1 S2 . . . Sd = S11 S12 . . . S1d

S21 S22 . . . S2d......

. . ....

Sm1 Sm2 . . . Smd

Rmd

Hedge an obligation X RmHave to pay Xi if state i occurs

Buy/short sell = (1, . . . , d) shares to cover obligation

2


15/23

Hedging problem (contd)

Position Rd purchased at time t= 0j = number of shares of asset j purchased, j = 1, . . . , d

Cost of the position = dj=1

pjj = p

Payoff from liquidating position at time t= 1

payoffyi in state i: yi = dj=1 SijjStacking payoffs for all states: y = SViewing the payoff vector y: y range(S)

y = S1 S2 . . . Sd12...d

=

d

j=1

jSj

Payoff y hedges X if y X.

3


16/23

Hedging problem (contd)

Optimization problem:

min dj=1 pjj ( p)subject to dj=1 Sijj Xi, i= 1, . . . , m ( S X)Features of this optimization problem

Linear objective function: p

Linear inequality constraints: S X

Example of a linear program

Linear objective function: either a min/max

Linear inequality and equality constraintsmax/minx c

xsubject to Aeqx = beq

Ainx bin

4


17/23

Linear programming duality

Linear programP = minx c

x

subject to Ax bDual linear program

D = maxu bu

subject to Au = cu

0

Theorem.

Weak Duality: P DBound: x feasible for P, u feasible for D, cx

P

D

bu

Strong Duality: Suppose P or D finite. Then P= D.

Dual of the dual is the primal (original) problem

5

M d l l


18/23

More duality results

Here is another primal-dual pair

minx c

xsubject to Ax = b = maxu b

usubject to Au = c

General idea for constructing duals

P = min{cx : Ax b} min{cx u(Ax b) : Ax b} for all u 0 bu + min{(c Au)x : x Rn}=

bu Au = c

otherwise max{bu : Au = c}

Lagrangian relaxation: dualize constraints and relax them!

6


19/23

Financial Engineering and Risk ManagementReview of nonlinear optimization



U t i d li ti i ti


20/23

Unconstrained nonlinear optimization

Optimization problemminxRn f(x)

Categorization of minimum pointsx global minimum if f(y) f(x) for all y

xloc local minimum if f(y) f(xloc) for all y such that y x

loc r

Sufficient condition for local min

gradient f(x) =

fx1...f

xn

= 0: local stationarity

Hessian 2f(x) = 2f

x21

2f

x1x2. . .

2f

x1xn

... ... . . . ...2f

xnx1

2f

xnx2. . .

2f

x2n

positive semidefiniteGradient condition is sufficient if the function f(x) is convex.

2

U t i d li ti i ti


21/23

Unconstrained nonlinear optimization

Optimization problem

minxR2 x2

1+ 3x1x2 + x

3

2

Gradient

f(x) =

2x1 + 3x23x1 + 3x

22

= 0 x = 0,

9

4

3

2

Hessian at x: H =

2 3

3 6x2

x = 0: H =

2 3

3 0

. Not positive definite. Not local minimum.

x = 9

43

2

: H =

2 3

3 9

. Positive semidefinite. Local minimum

3

L i th d


22/23

Lagrangian method

Constrained optimization problemmaxxR2 2 ln(1 + x1) + 4 ln(1 + x2),

s.t. x1 + x2 =12

Convex problem. But constraints make the problem hard to solve.

Form a Lagrangian function

L(x, v

) =2 ln

(1

+x1

) +4 ln

(1

+x2

) v

(x1

+x2

12

)

Compute the stationary points of the Lagrangian as a function of v

L(x, v) = 2

1+x1 v

4

1+x2

v = 0 x1 =2

v 1, x2 = 4

v 1

Substituting in the constraint x1 + x2 = 12, we get

6

v= 14 v= 3

7 x = 1

3

11

25

4

Portfolio Selection


23/23

Portfolio Selection

Optimization problem

maxx x

xVx

s.t. 1x = 1

Constraints make the problem hard!

Lagrangian function

L(x, v) = x

xVx

v(1x

1)

Solve for the maximum value with no constraints

xL(x, v) = 2Vx v1 = 0 x =1

2 V1( v1)

Solve for v from the constraint

1x = 1 1V1( v1) = 2 v= 1

V1

21V11

Substitute back in the expression for x

5

lecture slides linalg

Documents