Linear Algebra 1

Industrial AI Lab.

Prof. Seungchul Lee

Some materials from linear algebra review by Prof. Zico Kolter from CMU

Linear Equations

• Set of linear equations (two equations, two unknowns)

2

1 2

1 2

4 5 13

2 3 9

x x

x x

Linear Equations

• Solving linear equations– Two linear equations

– In a vector form, 𝐴𝑥 = 𝑏, with

3

1 2

1 2

4 5 13

2 3 9

x x

x x

1

2

4 5 13, ,

2 3 9

xA x b

x

Linear Equations

• Solving linear equations– Two linear equations

– In a vector form, 𝐴𝑥 = 𝑏, with

– Solution using inverse

4

1 2

1 2

4 5 13

2 3 9

x x

x x

1

2

4 5 13, ,

2 3 9

xA x b

x

1 1

1

Ax b

A Ax A b

x A b

Linear Equations

• Solving linear equations– Two linear equations

– In a vector form, 𝐴𝑥 = 𝑏, with

– Solution using inverse

– Don’t worry here about how to compute matrix inverse

– We will use a numpy to compute

5

1 2

1 2

4 5 13

2 3 9

x x

x x

1

2

4 5 13, ,

2 3 9

xA x b

x

1 1

1

Ax b

A Ax A b

x A b

Linear Equations in Python

6

1 2

1 2

4 5 13

2 3 9

x x

x x

1

2

4 5 13, ,

2 3 9

xA x b

x

System of Linear Equations

• Consider a system of linear equations

• Can be written in a matrix form as 𝑦 = 𝐴𝑥, where

7

Elements of a Matrix

• Can write a matrix in terms of its columns

• Careful, 𝑎𝑖 here corresponds to an entire vector 𝑎𝑖 ∈ ℝ𝑚

• Similarly, can write a matrix in terms of rows

• 𝑏𝑖 ∈ ℝ𝑛

8

Vector-Vector Products

• Inner product: 𝑥, 𝑦 ∈ ℝ𝑛

9

Matrix-Vector Products

• 𝐴 ∈ ℝ𝑚×𝑛, 𝑥 ∈ ℝ𝑛 ⟺ 𝐴𝑥 ∈ ℝ𝑚

• Writing 𝐴 by rows, each entry of 𝐴𝑥 is an inner product between 𝑥 and a row of 𝐴

10

Matrix-Vector Products

• 𝐴 ∈ ℝ𝑚×𝑛, 𝑥 ∈ ℝ𝑛 ⟺ 𝐴𝑥 ∈ ℝ𝑚

• Writing 𝐴 by columns, 𝐴𝑥 is a linear combination of the columns of 𝐴, with coefficients given by 𝑥

11

Symmetric Matrices

• Symmetric matrix:

𝐴 ∈ ℝ𝑛×𝑛 with 𝐴 = 𝐴𝑇

• Arise naturally in many settings

• For 𝐴 ∈ ℝ𝑚×𝑛,

𝐴𝑇𝐴 ∈ ℝ𝑛×𝑛 is symmetric

12

Norms (Strength or Distance in Linear Space)

• A vector norm is any function 𝑓:ℝ𝑛 ⟹ℝ with

• 𝑙2 norm

• 𝑙1 norm

• 𝑥 measures length of vector (from origin)

13

Norms in Python

14

Orthogonality

• Two vectors 𝑥, 𝑦 ∈ ℝ𝑛 are orthogonal if

• They are orthonormal if

15

and

Angle between Vectors

• For any 𝑥, 𝑦 ∈ ℝ𝑛,

• (unsigned) angle between vectors in ℝ𝑛 defined as

16

Angle between Vectors

• ȁ𝑥 𝑥𝑇𝑦 ≤ 0 defines a half space with outward normal vector 𝑦, and boundary passing through 0

17