from linear algebrato machine learning

Omar GutiérrezJuly 27, 2018

@trinogz

Overview

Motivation

Vectors

Examples

Conclusions

motivation

Motivation

∙ Linear algebra is important to understand machine learning.∙ As well as calculus, probability theory, and statistics.∙ It is rewarding to take the hard path to learn machinelearning (IMHO).

Learning from errors

Learning from errors

vectors

A vector is a collection of numbers

~a = a =

a1a2...an

Length of a vector

|a| =

√√√√ n∑i=0

a2i

Distance beetwen vectors

d(a,b) = ||a− b||

=

√√√√ n∑i=0

(ai − bi)2

Dot product

a · b =n∑i=0

aibi

= a0b0 + a1b1 + . . .+ anbnSo,

a · a = a0a0 + a1a1 + . . .+ anan= a20 + a21 + . . .+ a2n= |a|2

examples

The AI winter is coming

∙ Is really coming? No.∙ However, we already had an AI winter.∙ The research on neural nets was stopped for many years,after Minsky and Papert proved that a single layer perceptronwas not able to deal with the exclusive-or problem.

Toy dataset

b11 b12 1

b21 b22 0

b31 b32 1

. . . . . . . . .

bn1 bn2 1

Perceptron

1

x1

x2

...

xn

∑f (x)

0.5

0.1

0.1

0.1

x0 x1∑

f (x)

1 1 1× 0.5+ 1×−1 = −0.5 01 0 1× 0.5+ 0×−1 = 0.5 1

Linear Regression

40 50 60 70 80 90 100150

160

170

180

190

200

x

y

Linear Regression

∙ We want to calculate the intercept a and theslope b.

arg mina,b

∑i

(yi − (axi + b))2 = arg minw

||Xw − y||2

∙ The solution to this optimization problem is:

w∗ = (XTX)−1XTy .

conclusions

More topics we should check

∙ Gradient descent is a beautiful optimization algorithm,basically, we multiply matrices to many times.

∙ Eigenvectors and eigenvalues; some dimensionalityreduction techniques are based on eigendecomposition.

∙ Be aware that numerical instabilities can happen, and avoidthese ones.

References

∙ Mathematics for Machine Learning: Linear Algebra byCoursera.

∙ The Math of Intelligence by Siraj Raval.∙ Deep Learning Book by Bengio and Goodfellow, has achapter summarizing which linear algebra topics you needto learn neural networks.

Thank you.Questions?Comments?