introduction to neural networks - in.tum.de · introduction to neural networks freek stulp. 2...

Introduction to Neural Networks

Freek Stulp

2

Overview

Biological BackgroundArtificial NeuronClasses of Neural Networks

1. Perceptrons2. Multi-Layered Feed-Forward Networks3. Recurrent Networks

Conclusion

3

Biological BackgroundNeuron consists of:? Cell body? Dendrites? Axon? Synapses

Neural activation :? Throught dendrites/axon? Synapses have different

strengths

4

Artificial Neuron

aj Wji

Input links(dendrites)

Unit(cell body)

Output links(axon)

ai ai =

g(ini)ini =

? ajWji

5

Class I: Perceptron

a = g(in)

in = ? ajWj

a = g(-W0 + W1a1 + W2a2) g(in) = 0, in<01, in>0{

a1

a2

-1

Ij

a

O

W0

W1

W2

Wj

6

7

Learning in PerceptronsError often defined as:

E(W) = 1/2? d? D(td-od)2

Go towards the minimum error!

Update rules:? wi = wi ??? wi

? ? wi = -? ?E/?wi

? ?E/?wi = ?/?wi 1/2? d? D(td-od)2

= ? d? D(td-od)iid

This is called gradient descent

i

8

Class II: Multi-layer Feed-forward Networks

Feed-forward:? Output links only connected

to input links in the next layer

Input Hidden OutputMultiple layers:? hidden layer(s)

Complex non-linear functions can be represented

9

Learning in MLFF NetworksFor output layer, weight updating similar to perceptrons.Problem: What are the errors in the hidden layer?Backpropagation Algorithm? For each hidden layer (from output to input):

?For each unit in the layer determine how much it contributed to the errors in the previous layer.

?Adapt the weight according to this contribution

This is also gradient descent

10

Class III: Recurrent Networks

Input Hidden OutputNo restrictions on connections

Behaviour more difficult to predict/ understand

11

Conclusion

Inspiration from biology, though artificial brainsare still very far away.

Perceptrons too simple for most problems.

MLFF Networks good as function approximators.? Many of your articles use these networks!

Recurrent networks complex but useful too.

12

Literature

Artificial Intelligence: A Modern Approach? Stuart Russel and Peter Norvig

Machine Learning? Tom M. Mitchell