part iii cybernetics - monash universityusers.monash.edu/~app/cse2330/04/lnts/ch06.pdf · page...

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

1

Pa

rt I

II C

yb

ern

etic

s

C

hap

ter

6 C

on

cep

t N

euro

ns

Ch

ap

ter

7 N

eura

l C

od

ing

Ch

ap

ter

8 O

ur

Fri

end

th

e L

imu

lus

Ch

ap

ter

9 S

up

ervis

ed L

earn

ing

Ch

ap

ter

10 A

ssoci

ati

ve

Mem

ory

Net

work

s

Thes

e ch

apte

rs p

rese

nt

clas

sica

l ar

tifi

cial

neu

ral

net

work

s an

d t

he

subst

ance

this

cours

e is

conce

ntr

ated

on.

Tw

o c

hap

ters

are

lac

kin

g i

n t

he

book,

one

on U

nsu

per

vis

ed L

earn

ing a

nd o

ne

on

Rei

nfo

rcem

ent

Lea

rnin

g w

hic

h i

n b

rain

model

ling a

re m

uch

more

im

port

ant

than

super

vis

ed

lear

nin

g, as

I s

ee i

t. W

e w

ill,

of

cours

e, d

iscu

ss t

hes

e to

pic

s to

o.

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

2

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s T

he

McC

ull

och

-Pit

ts n

euro

n f

rom

1943 a

ccord

ing t

o L

ytt

on:

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

3

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s W

e w

ill

dra

w n

euro

ns

as b

elow

The

synap

ses

are

dra

wn a

s ar

row

s, w

hic

h i

s our

sym

bol

for

exci

tato

ry s

ynap

ses.

The

wei

ghte

d s

um

of

the

affe

rents

is

call

ed a

ctiv

atio

n p

ote

nti

al.

The

nonli

nea

rity

, w

hic

h i

s a

par

t of

alm

ost

all

neu

ron m

odel

s, r

esid

es i

n t

he

circ

le.

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

4

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s

A b

inar

y n

euro

n w

ill

rece

ive

bin

ary

inputs

fro

m o

ther

bin

ary

neu

rons.

The

inputs

coll

cted

in

vec

tors

can

be

illu

stra

ted i

n t

wo a

nd t

hre

e dim

ensi

ons.

Fro

m t

her

e on w

e m

ust

gen

eral

ize.

Ly

tton, fi

g 6

.6

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

5

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s

A “

com

pu

tin

g”

neu

ron

(L

ytt

on,

fig

6.2

):

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

6

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s R

eal

neu

ron

s h

ave

a sm

oo

th n

onli

nea

rity

(L

ytt

on

, fi

g 6

.3):

Ther

e ar

e se

ver

al d

iffe

rent

way

s to

des

crib

e th

e nonli

nea

rity

mat

hem

atic

ally

, th

e m

ain d

iffe

rence

bei

ng c

onven

ience

.

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

7

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s

Dif

fere

nt

neu

ron

s ar

e co

nn

ecte

d i

n d

iffe

ren

t d

irec

tion

s in

dif

fere

nt

cort

ical

lay

ers:

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

8

(Fro

m K

olb

& W

his

haw

: F

undam

enta

ls o

f H

um

an N

euro

psy

cholo

gy)

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s

We

shou

ld a

lso

rem

emb

er t

hat

th

ere

is a

co

lum

nar

org

aniz

atio

n (

Fro

m M

oun

tcas

tle:

”Th

e co

lum

nar

org

aniz

atio

n o

f th

e n

eoco

rtex

”, B

rain

199

7, p

p. 7

01

-72

2):

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

9

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s

In a

rtif

icia

l n

eura

l n

etw

ork

s w

e h

ave

gen

eric

neu

ron

s, c

on

nec

ted

in

ver

y

sim

pli

fied

cir

cuit

s.

Dep

end

ing

on

th

e m

ain

fea

ture

s re

tain

ed i

n t

he

arti

fici

al n

eura

l n

etw

ork

s w

e

hav

e m

ain

ty

pes

of

net

wo

rks,

su

ch a

s fe

edfo

rwa

rd n

etw

ork

s an

d r

ecu

rren

t

net

wo

rks.

(in

Sec

tio

n I

V B

rain

s w

e w

ill

mak

e se

rio

us

atte

mp

ts t

o i

mp

rov

e o

ur

mo

del

s)

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

10

C

ha

pte

r 6

, C

on

cep

t N

euro

ns

A

one

lay

er f

eedfo

rwar

d n

etw

ork

is

show

n b

elow

Just

one

lay

er o

f neu

rons

wit

h n

o i

nte

rdep

enden

ce –

not

much

of

a net

work

yet

.

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

11

C

ha

pte

r 6,

Co

nce

pt

Neu

ron

s

A t

wo

lay

er f

eed

forw

ard

art

ific

ial

neu

ral

net

wo

rk (

Ly

tto

n f

ig 6

.7)

Th

ere

is a

hid

den

lay

er “

bef

ore

” th

e o

utp

ut

lay

er o

f n

euro

ns.

Ly

tto

n c

alls

th

is a

th

ree

lay

er n

etw

ork

, co

un

ting

th

e in

pu

ts a

s a

lay

er.

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

12

C

ha

pte

r 6,

Co

nce

pt

Neu

ron

s

A p

arti

ally

connec

ted f

eedfo

rwar

d n

etw

ork

(f

rom

Hay

kin

: N

eura

l N

etw

ork

s A

Com

pre

hen

sive

Foundat

ion)

This

is

alre

ady

a b

iolo

gic

ally

more

rea

list

ic m

odel

than

the

full

connec

tivit

y m

odel

s.

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

13

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s A

rec

urr

ent

net

work

is

show

n b

elow

. T

he

effe

rent

signal

s ar

e fe

d b

ack t

o b

ecom

e par

t of

the

affe

rent

signal

s. T

her

e ar

e lo

ops

in r

ecurr

ent

net

work

s! T

he

fill

ed c

ircl

es a

re i

nhib

itory

sy

nap

ses.

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

14

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s

A r

ecu

rren

t n

etw

ork

of

neu

ron

s (L

ytt

on, fi

g 6

.4).

Th

ere

are

no

aff

eren

ts i

n t

his

ver

sio

n.

Len

nar

t G

ust

afss

on pag

e

Cours

e note

s C

SE

2330

15

Ch

ap

ter

6,

Co

nce

pt

Neu

ron

s

Sta

tes

kee

p c

ha

ng

ing

, at

lea

st f

or

som

e ti

me,

in

a r

ecu

rren

t n

etw

ork

(Ly

tto

n,

fig

6.5

; th

ere

is a

mis

tak

e in

th

e b

oo

k –

un

it a

sh

ou

ld a

lway

s b

e 0

):