Neuronal basis of natural textures coding

in area V4 of the awake monkey: texture analysis

P.Girard, C. Jouffrais, F. Arcizet, J. Bullier

Insight2+IST–2000-296883D shape and material properties for recognition

Aim of the study (WP3)

Coding of material properties

In area V4 of awake macaque monkey

Performing a visual fixation task

Stimuli from the CURET database:12 textures + 12 scrambled textures

Frontal viewing direction

3 illumination directions (22.5, 45 and 67.6 deg.)

72 stimuli

Stimuli

Terrycloth Sand paper Plaster Aluminum foil

Salt crystalsRoof shinglePlaster (zoom)Lettuce leaf

Linen Concrete White bread Soleirolia plant

Experimental setup

Control of the experiment and real time analog and digital acquisition: CORTEX (courtesy of NIH)

5 independent microelectrodes (TREC)

Sorting software: MSD (Alpha-Omega)

Eye monitoring: IScan eye-tracker (120 Hz, 0.2 DVA)

Protocol

Mapping of the Receptive Field (RF)Hand-moved bars

M-sequences of black and white dots

Recording of response to the 72 stimuli (10 trials per stimulus)

Control: 36 original textures moved 1 deg apart

Recording sites

.

5°

10°

F ovea10° 5°

Database and statistics

Database:167 cells (42 with unshaped stimuli, 98 with shaped stimuli, 27 with new set of textures)

StatisticsANOVA 3-factors (Texture, Illum. Dir., Type)

Population (Rank analysis, MDS, comparison V4/IT)

V4 neuron sharply selective to textures

-0.5 0 0.5 1 1.50

1

-0.5 0 0.5 1 1.50

1

-0.5 0 0.5 1 1.50

1

-0.5 0 0.5 1 1.5

-0.5 0 0.5 1 1.5

-0.5 0 0.5 1 1.50

0.2

0.4

-0.5 0 0.5 1 1.5

-0.5 0 0.5 1 1.5

-0.5 0 0.5 1 1.50

0.2

-0.5 0 0.5 1 1.5

-0.5 0 0.5 1 1.5

-0.5 0 0.5 1 1.50

0.2

Plaster (zoom)Lettuce leaf

0.5s

100

0

Sp

ikes/

s

On Off

22.5 deg.

45 deg.

67.6 deg.

Texture

neuron selective to illumination direction

Example of a V4 cell whose discharge is systematically increased for a lighting direction of 67.6 deg.

22.5 deg.

45 deg.

67.6 deg.

Illum. dir.Error Bars show 95.0% Cl of Mean

Alu

min

um

Bre

ad

Con

cre

te

Leaf

Lin

en

Pla

nt

Pla

ster

Pla

ster

(z)

Sal

t

San

dpa

per

Shi

ngle

Terr

ycl

oth

Texture

0

20

40

60S

pik

es/s

.

V4 neuron selective to original and “moved” textures

Example of a V4 cell whose selectivity is the same for ‘original’ and ‘moved’ conditions. No response to scrambled sitmuli.

moved

original

scrambled

StimuliError Bars show 95.0% Cl of Mean

Alu

min

um

Bre

ad

Con

cre

te

Leaf

Lin

en

Pla

nt

Pla

ster

Pla

ster

(z)

Sal

t

San

dpa

per

Shi

ngle

Terr

ycl

oth

Texture

-10

0

10

20

30

Sp

ikes

/s.

Statistics

3 factors ANOVA (main effect + interaction, P<0.05) shows that:

82% of the cells are selective to textures

69% of the cells have a different response to original and random-phase textures

69% of the cells are selective to lighting direction

82% selective to texture

Multidimensional Scaling (MDS) – originals

Dim

ensi

on 2

MDS analysis performed on 68 cells. Original textures only, final configuration, 3 dimensions (Alienation:0.108, Stress: 0.099).

Correlations of neuronal responses with first,second,third and fourth order parameters

Median luminance

Rms contrast

skewness

kurtosis

Texture analysis

Is there a match between V4 cell population and a set of filters that could be used to classify the textures?

Are there other interesting parameters that characterize the textures and are coded in V4?

Texture analysis: methodology

Sets of 2D GABOR filters (several sizes, spatial frequencies and 8 orientations (0°:22.5:157.5°)

3 different types of quantification of outputs- thresholds

-energy

-Spectral histograms

Example of filter and computations (thresholds)

Size= 12 pixels, freq: 9.5 c/°, sigma 4 pixels, orientation 0

Size= 12 pixels, freq: 14 c/°, sigma 4 pixels, orientation 0

Example of cluster analysis with filters and neurons

Example of filter and computations (energy)

Cluster analysis based upon energy

N=56

filters: Size 12 pixels, freq: 2 to 28 c/°, sigma 3 pixels, orientations 0:22.5:157.5°

MDS based upon energy

Spectral Histogram

N=29

C o m p a r i s o n o f M S D a n a l y s i s f o r f i l t e r s ( p a r a m e t e r s : s i z e = 1 2 , 2 4 , 3 6 p i x e l s , f r e q u e n c y = 4 . 7 6 c y c l e s / d e g , 8 o r i e n t a t i o n s ( N = 2 9 , c e n t r a l

s t i m u l i , t e x t u r e s * p < 0 . 0 0 1 )

a n d c e l l s .

filters (parameters: size=12, 24, 36 pixels, frequency= 4.76 cycles/deg,8 orientations and cells (N=29, central stimuli, textures* p<0.001).

Spectral Histogram vs ENERGY

energy Spectral histogram

MDS over different epochs after the stimulus onset filters: Size 12 pixels, freq: 2 to 28 c/°, sigma 3 pixels, orientations 0:22.5:157.5°

MDS with images (filters/neurons)

New textures

SNR is an important parameterMean2/std2 (of image, not of filtered image)

Snr : 1 possible dimensionN=27

filters: Size 12 pixels, freq: 2 to 28 c/°, sigma 3 pixels, orientations 0:22.5:157.5°

SNR another example

Mds with images of the textures

Luminance?

Conclusions

Coding of material properties in V4 and IT

Is this indeed texture classification or identification? We need expert advice to use better texture characterization (Spatial frequency…) or classification (Varma and Zisserman, Geusebroek and Smeulders)

Do neurons perform such expert classification?

Need to use a comparable behavioural task?

Not shown