low level visual processing. information maximization in the retina hypothesis: ganglion cells try...
TRANSCRIPT
![Page 1: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/1.jpg)
Low Level Visual Processing
![Page 2: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/2.jpg)
Information Maximization in the Retina
• Hypothesis: ganglion cells try to transmit as much information as possible about the image.
• What kind of receptive field maximizes mutual information I(s,r)?
![Page 3: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/3.jpg)
Information Maximization in the Retina
• In this particular context, information is maximized for a factorial code:
• For a factorial code, the mutual information is 0 (there are no redundancies):
1
n
ii
P P r
r
1 2( , ,..., ) 0nI I r r r r
![Page 4: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/4.jpg)
Information Maximization in the Retina
Independence is hard to achieve. Instead, we can look for codes that decorrelate the activity of the ganglion cells. This is a lot easier because decorrelation can be achieved with a simple linear transformation.
![Page 5: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/5.jpg)
Information Maximization in the Retina
We assume that ganglion cells are linear:
The goal is to find a receptive field profile, Ds(x), for which the ganglion cells are decorrelated (i.e., a whitening filter).
*
s s
s
L a dxD x a s x
D s a
![Page 6: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/6.jpg)
Information Maximization in the Retina
Intuition: take a white noise image and filter it with a gaussian filter. To decorrelate the resulting image, you need to deconvolve. The deconvolving filter would be a mexican hat (band pass) filter in this case.
![Page 7: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/7.jpg)
Information Maximization in the Retina
• Correlations are captured by the crosscorrelogram (all signals are assumed to be zero mean):
,LL LL s sQ a b Q a b L a L b
• The crosscorrelogram is also a convolution (in 4D)
s sdxdyD x a D y b s x s y
![Page 8: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/8.jpg)
Fourier Transform
• The Fourier transform of a convolution is equal to the product of the individual spectra:
• The spectrum of a Dirac function is flat.
*h x f g x f x g d
H F G
![Page 9: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/9.jpg)
Information Maximization in the Retina
• To decorrelate, we need to ensure that the crosscorrelogram is a Dirac function, i.e., its Fourier transform should be as flat as possible.
2LL LQ a b a b
2LL LQ k
![Page 10: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/10.jpg)
Information Maximization in the Retina
2s s LdxdyD x a D y b s x s y a b
2 2s ss LD k Q k
Ls
ss
D kQ k
![Page 11: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/11.jpg)
Information Maximization in the Retina
2 20
k
ss
eQ k
k k
Ls
ss
D kQ k
2 20
kLe k k
Grows with k.. Amplifies noise!
![Page 12: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/12.jpg)
Information Maximization in the Retina
• If we assume that the retina adds noise on top of the signal to be transmitted to the brain, the previous filter is a bad idea because it amplifies the noise.
2E dk D k s k k s k
D k
• Solution: use a noise filter first, :
![Page 13: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/13.jpg)
Information Maximization in the Retina
ss
ss
Q kD k
Q k Q k
Ls
ss
D k D kQ k
L ss
ss
Q k
Q k Q k
Goes to 0 as k goes to infinity
![Page 14: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/14.jpg)
Information Maximization in the Retina
• The shape of the whitening filter depends on the noise level.
• For high contrast/low noise: bandpass filter. Center-surround RF.
• For low contrast/high noise: low pass filter. Gaussian RF
![Page 15: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/15.jpg)
Information Maximization in the Retina
j
+
![Page 16: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/16.jpg)
Information Maximization in the Retina
0 00 0
55 10 1015
1 1
2 2
3 3
44
temporal frequency (Hz) temporal frequency (H
![Page 17: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/17.jpg)
Information Maximization beyond the Retina
• The bottleneck argument can only work once…
• The whitening filter only decorrelates. To find independent components, use ICA: predicts oriented filter
• Use other constrained beside infomax, such as sparseness.
![Page 18: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/18.jpg)
Information Maximization beyond the Retina
• One big problem with the previous theory: Receptive field size goes down for very low luminance level…
• Alternative: the retina tries to transmit edges.
![Page 19: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/19.jpg)
Center Surround Receptive Fields
• The center surround receptive fields are decent edge detectors
+
![Page 20: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/20.jpg)
Center Surround Receptive Fields
![Page 21: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/21.jpg)
Center Surround Receptive Fields
• At high luminance: small RFs are fine because the SNR is high
• At intermediate luminance: large RFs are needed for signal averaging
• At very low luminance, larger RF would not be good because most of the info would be lost. Use small RFs instead and look for support in the cortex for contiguous contours
![Page 22: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/22.jpg)
Feature extraction: Energy Filters
![Page 23: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/23.jpg)
2D Fourier Transform
x
y
![Page 24: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/24.jpg)
2D Fourier Transform
Frequency
Orientationx
y
![Page 25: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/25.jpg)
2D Fourier Transform
x
y
![Page 26: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/26.jpg)
Motion Energy Filters
Space
Tim
e
![Page 27: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/27.jpg)
Motion Energy Filters
• In a space time diagram 1st order motion shows up as diagonal lines.
• The slope of the line indicates the velocity
• Therefore, a space-time Fourier transform can recover the speed of motion
![Page 28: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/28.jpg)
Motion Energy Filters
![Page 29: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/29.jpg)
Motion Energy Filters
![Page 30: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/30.jpg)
![Page 31: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/31.jpg)
Motion Energy Filters
• 1st order motionT
ime
Space
![Page 32: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/32.jpg)
Motion Energy Filters
• 2nd order motion
http://www.psypress.com/mather/resources/swf/Demo11_9.swf
![Page 33: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/33.jpg)
Motion Energy Filters
• In a space time diagram, 2nd order motion does not show up as a diagonal line…
• Methods based on linear filtering of the image followed by a nonlinearity cannot work
• You need to apply a nonlinearity to the image first
![Page 34: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/34.jpg)
Motion Energy Filters
• A Fourier transform returns a set of complex coefficients:
( )
cos sin
ji x
j
j j j
j j j
I x c e
c x i x
c a ib
![Page 35: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/35.jpg)
Motion Energy Filters
• The power spectrum is given by
2 2 2
2 2 2 2
,
cos sin
cos , sin
j
j j j j j j
i
j j j j j
j jj j
j j j j
c a ib c a b
c c e c i
a b
a b a b
2
jc
![Page 36: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/36.jpg)
Motion Energy Filters
( )
( ) cos sin
( )cos
( )sin
ji x
j
j j
j j
j j
j j
c I x e dx
I x x i x dx
a ib
a I x x dx
b I x x dx
![Page 37: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/37.jpg)
Motion Energy Filters
2 2 2
( )cos
( )sin
j j
j j
j j j
a I x x dx
b I x x dx
c a b
cos
sin+
aj
bj
cj2x2
x2
I
![Page 38: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/38.jpg)
Motion Energy Filters
• Therefore, taking a Fourier transform in space time is sufficient to compute motion. To compute velocity, just look at where the power is and compute the angle.
• Better still, use the Fourier spectrum as your observation and design an optimal estimator of velocity (tricky because the noise is poorly defined)…
![Page 39: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/39.jpg)
Motion Energy Filters
• How do you compute a Fourier transform with neurons? Use neurons with spatio-temporal filters looking like oriented sine and cosine functions.
• Problem: the receptive fields are non local and would have a hard time dealing with multiple objects in space and multiple events in time…
![Page 40: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/40.jpg)
Motion Energy Filters
• Solution: use oriented Gabor-like filters or causal version of Gabor-like filters.
• To recover the spectrum, take quadrature pairs, square them and add them: this is what is called an Energy Filter.
2
2( ) exp sin2x
f x x
![Page 41: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/41.jpg)
Motion Energy Filters
x ( )cosj ja I x x dxx2
Local cosine filter: Gabor filter
![Page 42: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/42.jpg)
Motion Energy Filters
( )sinj jb I x x dxx x2
Local sine filter: Gabor filter
![Page 43: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/43.jpg)
Motion Energy Filters
x2
x2
+
Neurons do not have negative activity: use quadrature pairs
![Page 44: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/44.jpg)
V1 energy filters
t
x
Constant velocity
![Page 45: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/45.jpg)
t
x
![Page 46: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/46.jpg)
Bank of filters
y
x
![Page 47: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/47.jpg)
From V1 to MT
• V1 cells are tuned to velocity but they are also tuned to spatial and temporal frequencies
t
x
2
2( , ) exp sin2 x t
x tr x t x t
![Page 48: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/48.jpg)
From V1 to MT
t
x
Constant velocity
![Page 49: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/49.jpg)
From V1 to MT
• MT cells are tuned to velocity across a wide range of spatial and temporal frequencies
t
x
![Page 50: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/50.jpg)
MT Cells
![Page 51: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/51.jpg)
Pooling across Filters
• Motion opponency: it is not possible to perceive transparent motion within the same spatial bandwidth. This suggests that the neural read out mechanism for speed computes the difference between filters tuned to different spatial frequencies within the same temporal bandwidth.
![Page 52: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/52.jpg)
Pooling across Filters
t
+ Flicker x
![Page 53: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/53.jpg)
Energy Filters
• For second order motion, apply a nonlinearity to the image and then run a motion energy filter.
![Page 54: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/54.jpg)
Energy Filters: Generalization
• The same technique can be used to compute orientation, disparity, … etc.
![Page 55: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/55.jpg)
Energy Filters: Generalization
• Stereopsis: constant disparities correspond to oriented lines in right/left RF diagram.
![Page 56: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/56.jpg)
Energy Filters: Generalization
-50 0 50-1
-0.5
0
0.5
1
20 40 60 80 100
20
40
60
80
100
20 40 60 80 100
20
40
60
80
100
10 20 30 40
10
20
30
40
![Page 57: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/57.jpg)
Energy Filters: Generalization
![Page 58: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/58.jpg)
![Page 59: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/59.jpg)
Orientation Selectivity
• At first sight: a simple, if not downright stupid, problem.
• Use an orientation energy filter
• The only challenge: finding out exactly how the brain does it…
• Two classes of models: feedforward and lateral connection models.
![Page 60: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/60.jpg)
Orientation Selectivity: Feedforward (Hubel and Wiesel)
Pooled LGN afferences
LGN
Cortex
Orientation (deg)
-45 0 450
20406080
100
Act
ivity
spi
kes/
s
Orientation (deg)
-45 0 450
2468
10
Pooled LGN Tuning Curves
![Page 61: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/61.jpg)
Orientation Selectivity: The Lateral Connection Model
Pooled LGN afferences
Output Tuning Curve
LGN
Cortex
Orientation (deg)
-45 0 450
20406080
100
Act
ivity
spi
kes/
s
Orientation (deg)
-45 0 450
2468
10
Pooled LGN Tuning Curves
![Page 62: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/62.jpg)
Orientation Selectivity
• How does it work? Simple consequence of Fourier transform…
![Page 63: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/63.jpg)
Orientation Selectivity: Feedforward (Hubel and Wiesel)
--+
![Page 64: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/64.jpg)
Orientation Selectivity
• To get a complex cell: take a quadrature pair, rectify, square and sum.
![Page 65: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/65.jpg)
Orientation Selectivity
• The evidence in favor for the Hubel and Wiesel model are overwhelming– No further tuning over time– Aspect ratio consistent with this model– LGN input to layer 4 cells as tuned as output– LGN/Cortex connectivity as predicted by
feedforward model
• Why are there lateral connections?
![Page 66: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/66.jpg)
Orientation Selectivity
• But which model is the most efficient?
![Page 67: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/67.jpg)
Fisher information
• Fisher information, one neuron:
• For a population of neurons with independent noise:
2
2
fI
xyu
T rrw w
xyu
T rrw w
2
21 1
N Ni
ii ii
fI I
![Page 68: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/68.jpg)
Current view
Common conclusion:
Increasing the slope leads to better discrimination (or more information)
Two ways to increase the slope:
1. Sharpening
2. Increase in gain
xyu
T rrw w
xyu
T rrw w
2
21
Ni
i i
fI
![Page 69: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/69.jpg)
Main Problems
1- Neurons are not independent
2
21
Ni
i i
fI
xyu
T rrw w
1 1 1trTI f Q f' Q Q Q Q
Q: Covariance matrix of the noise
![Page 70: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/70.jpg)
Source of the variability
Where does the noise come from?
• The variability comes from the dynamics of balanced recurrent networks
xyu
T rrw w
xyu
T rrw w
![Page 71: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/71.jpg)
Source of the variability
• Tuning curves and variability are intertwined
xyu
T rrw w
• So what?
f1()
f2()
f3()
Changing tuning curves requires changing the synaptic weights
Changing the synaptic weights changes the noise
distribution
![Page 72: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/72.jpg)
Orientation Selectivity
Output Tuning Curve
Cortex
Orientation (deg)
-45 0 450
20406080
100
Act
ivity
spi
kes/
s
Pooled LGN Tuning Curves
Output Tuning Curve
LGN
Cortex
Orientation (deg)
-45 0 450
20406080
100
Act
ivity
spi
kes/
s
Orientation (deg)
-45 0 450
2468
10
Pooled LGN Tuning Curves
Orientation (deg)
-45 0 450
2468
10
No-Sharpening Sharpening
LGN
2- Is this a bad representation of orientation?
2- Does sharpening help?
1- Are the output representations equally informative?1- Are the output representations equally informative?
3- Does sharpening help?
![Page 73: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/73.jpg)
Orientation Selectivity
• The no-sharpening model extracts a lot more information
xyu
T rrw w
0
1
2
3
4
5
6
NS S
Info OUT
Info
rmat
ion
(deg
-2)
![Page 74: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/74.jpg)
Orientation Selectivity
• Why are the models extracting different amount of information? It’s all in the output covariance matrices…
xyu
T rrw w
xyu
T rrw w
SNS
180
0
90180
90Preferred Orientation (deg) Preferr
ed O
rientat
ion (deg
)180
0
90180
90Preferred Orientation (deg) Preferr
ed O
rientat
ion (deg
)
![Page 75: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/75.jpg)
Conclusions
• Fisher information: what matters is the ratio of the slope over the variance of the noise
• However, in the presence of correlations, increasing the slope does not necessarily translate into better representations.
![Page 76: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/76.jpg)
Object Recognition
![Page 77: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/77.jpg)
Nature of the problem
• Given an image, recognize (identify) the objects that are currently seen.
• The hard part: the same object can look incredibly different in different images due to differences in view points, illumination, color, occlusions,… etc.
• A robust recognizer must develop invariant recognition (translation, size, rotation?)
![Page 78: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/78.jpg)
What do we know from imaging
• Are they specialized areas? Face and scene areas? Or areas for expertise?
• How distributed are the representations?
![Page 79: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/79.jpg)
Match Filter/Nearest Neighbor
• Store images of the objects under different viewpoint, scale, orientation, etc.
• To recognize, find the template that matches the image the best.
• Optimal under a gaussian noise assumption which is clearly wrong in this case….
• Not invariant • Works surprisingly well… (especially with
tangent distance)
![Page 80: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/80.jpg)
Object as nonlinear manifolds
• The set of points corresponding to the same object forms a nonlinear manifold in pixel space (of whatever space is used to represent the object)
• Tangent distance: compute distance to the tangent plane to the manifold.
• Tangent distance is not unlike relaxation in line attractor networks…
![Page 81: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/81.jpg)
Wavelets Templates
• Filter the image with wavelets
• Store objects as vector of wavelets activation
• Slightly more invariant
![Page 82: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/82.jpg)
Radial Basis Functions
• A set of functions is said to form a basis set if most other functions can be approximated as linear combinations of the basis functions.
• Ex: sine and cosine functions (Fourier transform) or gaussians (RBF).
1
n
i ii
f x c B x
![Page 83: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/83.jpg)
Radial Basis Functions
-100 0 1000
50
100
150
200
250
Direction (deg)
Act
ivity
-200 -100 0 100 2000
0.2
0.4
0.6
0.8
1
Preferred Direction (deg)
Act
ivity
![Page 84: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/84.jpg)
Radial Basis Functions
• Paper clip recognition. Parameterize the object with the angles between adjacent segments.
• Recognition is a nonlinear mapping from angle space onto identity.
• Use an RBF network to implement the mapping
![Page 85: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/85.jpg)
Radial Basis Functions
Object 1
Object 2
![Page 86: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/86.jpg)
Radial Basis Functions
Object 1
View 1
View Based Unit
Invariant unit forObject 1
![Page 87: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/87.jpg)
Radial Basis Functions
Object 1
View 1
2 2 2
1 1 2 2 3 3
1 2 3 2, , exp
2
i i i
ia
![Page 88: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/88.jpg)
Radial Basis Functions
• RBF network: not unlike nearest neighbours but can be applied to high level features, not just pixel values
• Illustrate the value of overcomplete set: good recognition can be performed with a simple linear computation
![Page 89: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/89.jpg)
Radial Basis Functions
• Learning: use a gradient descent procedure to tune the network parameters
• The RBF units can be tuned with an unsupervised learning rule.
![Page 90: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/90.jpg)
Radial Basis Functions
• Problems: how does the cortex extract angles?
• How does this generalize to other objects?
![Page 91: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/91.jpg)
Vie w-tune d c e lls
M AX
we ig hte d su m
Sim p le c e lls (S1)
C o m p le x c e lls (C 1)
C o m p le x c o m p o site c e lls (C 2)
C o m p o site fe a ture c e lls (S2)
![Page 92: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/92.jpg)
Radial Basis Functions
• Basis sets can be complete (sine) or overcomplete (Wavelets, RBF).
![Page 93: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/93.jpg)
Radial Basis Functions
• Representation using complete sets are often very distributed (e.g. Fourier transform) and global. This means that they can be hard to decode and poor at dealing with multiple objects
• Overcomplete sets are memory intensive but they lead to sparse representations (linearly decodable), they work with multiple objects and they are efficient for learning.
![Page 94: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/94.jpg)
Convolution Networks
LeNet• Multilayer networks with hierarchical organization
and built-in translation invariance • Fine to coarse analysis provide some scale and
rotation invariance, plus noise resistance.• No natural description of the features used for the
decomposition… Bad news for neurophysiologists…
![Page 95: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/95.jpg)
Geons
• Two ideas:– Objects can be decomposed in terms of
combinations of simple shapes (geons)– Object representations involve a list of geons
and relative position in object-centered coordinates
• Finally an approach that does not turn the brain into a lookup table…
![Page 96: Low Level Visual Processing. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the](https://reader035.vdocuments.mx/reader035/viewer/2022062321/56649e445503460f94b38373/html5/thumbnails/96.jpg)
Geons
• Numerous problems:– No convincing experimental support and lots of
data against it – No good theory on how to build such
representations (current implementations cannot deal with real images)
– The list of Geons seems awfully arbitrary…