Mathematics for Machine LearningSpecial Mathematics LectureNagoya University, Fall 2020

https://www.henrikbachmann.com/mml_2020.html

Lecture 11: Neural Networks I

Machine learning overview

Remaining plan:• Introduce (simple) Neural Networks (Today)• Understand how to train them (Next time)• Consider more complicated NN.• Use finished implementations, e.g. Tensorflow.

Nice to watch

https://www.3blue1brown.com/neural-networks

https://nnfs.io/

3blue1brown

Neural Networks from Scratch

Neural Networks

Recall: Logistic regression

Logistic regression

Hypothesis:

Example: Binary classifier (Pass an exam Yes/No, Spam email Yes/no)

We learned the correct parameters by maximizing the log-likelihood (by using gradient ascent)

Or mizimizing the negative of the log-likelihood (= cost function) (by using gradient descent)

c galt

S

Oot.fit tOdXd

lXEIRdXo I

oft

Rewriting logistic regression as a neural network

Neuron linearpartInput f activation

stürmtlinear 5 activation R

weight Part function

WEIRbFpbias

oldNotationdb Oo

Motivation: Image classification

Goal: Check if a picture contains a cati upper index layer ilower index neuron

Picture Z linearpart a Output100

100 D ü dumb

Pixel 3 colorsEtwas 9

A 1001003

Motivation: Image classification

Goal: Check if a picture contains a cat or dog

Motivation: Image classification. More layers

Goal: Check if a picture contains a cat or dog as Hidden layers

WP

i

iii iii

id

(rough) Notation

O

Content of a layer

weil Iii

III i am

Content of a layer

Example of activation functionsThere are several common activiation functions.

https://en.wikipedia.org/wiki/Activation_function

Question: Why not use the identity map as an activation function?F

Example of NNLet us consider a NN with 4 layers (2 hidden):

I IRT

ER

amwaz.pk EEtRybIp3a

faftlIIYI Köln b