Latency, duration and codes for objectsin inferior temporal cortex

Gabriel Kreiman, Chou Hung, Tomaso Poggio, James DiCarlo

Center for Biological and Computational LearningComputation and Systems Biology Initiative

McGovern Institute for Brain Research

Massachusetts Institute of Technology

Recognition can be very fast

Stimulus presentation

100 ms100 ms

time

Passive viewing(fixation task)

5 objects presentedper second

•Recordings of spiking activity from macaque monkeys

•Recordings in an area involved in object recognition (inferior temporal cortex)

•10-20 repetitions per stimulus

• Presentation order randomized

• 77 stimuli drawn from 8 pre-defined categories

Recordings made by Chou Hung and James DiCarlo

Reading the neuronal code

1 2 3 4 5

Neuron 1 Neuron 2 Neuron 3 Object

Yes No No 1

Neuron 1 Neuron 2 Neuron 3 Object

Yes No No 1Yes Yes No 2Yes Yes Yes 3

Neuron 1 Neuron 2 Neuron 3 Object

Yes No No 1

Yes Yes No 2

Mind reading Neuronal readingCan we read out what the monkey is seeing?

xLearning from (x,y) pairs y ∈ {1,…,8}

Input to the classifier

100 ms0 ms 200 ms 300 ms

w

MUA: spike counts in each bin

LFP: power in each bin

MUA+LFP: concatenation of MUA and LFP

Accurate read-out of object category and identity from a small population

Hung*, Kreiman*, Poggio, DiCarlo. Science 2005

Decoding requires very few spikes

w = 12.5 ms

Hung*, Kreiman*, Poggio, DiCarlo. Science 2005

Local field potentials (the “input”) also show selectivity

Kreiman*, Hung*, Kraskov, Quiroga, Poggio, DiCarlo. Neuron 2006

MUA: multi-unit spiking activity

SUA: single-unit spiking activity

LFP: local field potentials

MUA & LFP: MUA combined with LFP

The classifier extrapolates to new scales and positions

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7 n=31 n=24 n=9 n=9 n=17 n=10

chance

TRAIN

TEST

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7 n=31 n=24 n=9 n=9 n=17 n=10

chance

TRAIN

TEST

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7 n=31 n=24 n=9 n=9 n=17 n=10

chance

TRAIN

TEST

Other observations

• We can decode information from local field potentials. MUA+LFP > MUA > LFP

• Feature selection significantly improves performance. Choosing the “best” neurons >> randomly selecting neurons

• We can decode the time of stimulus onset

• We can also read out coarse “where” information

• Decoding is robust to internal and external perturbations

• The population can extrapolate to novel pictures within known categories

We can decode object information from the model units

Serre, Kouh, Cadieu, Knoblich, Kreiman, Poggio. MIT AI Memo 2005

The model shows scale and position invariance

Latency, duration and codes for objectsin inferior temporal cortex

Gabriel Kreiman, Chou Hung, Tomaso Poggio, James DiCarlo

Center for Biological and Computational LearningComputation and Systems Biology Initiative

McGovern Institute for Brain Research

Massachusetts Institute of Technology