chapter 4 cortical memory - university of texas at austindana/ch4.pdf · cortical memory train of...

21
Chapter 4 Cortical Memory The cortical memory appears to compute in time steps of about 200 millisec- onds. During this time the memory settles to a state or decision that is used to guide the next action. The decision may be thought of as a branch in a program. The action may be overt as in reaching for a target, or covert as in simulating the reach internally. States of the memory are coded hierarchically with high level neurons coding abstract features and low-level neurons coding concrete features of the sensory surround. 4.1 Cortical Memory is Different One of the most difficult things to come to grips with is that the basic machinery of the brain seems to work in a way that is nothing like our conscious perception. Take a moment and look around you taking special care to tune in to the seemlessness of conscious experience. The your perception of the world seems clear, unaffected by the inhomogeneous resolution of the retina which provides clear resolution only inside a tiny degree of visual angle. Colors seem to be everywhere in your visual field even though the color measuring cones are concentrated need the center of the visual field. And the world seems stable and continuous even though it is sampled very discontinuously via discrete gaze points. How could this continuous perception happen? One constraint that op- erates is that the experience of conscious perception is slow compared to the brains mechanisms. Just generating a sentence takes several seconds whether audibly for external consumption or internal as part of our own 1

Upload: others

Post on 05-Oct-2020

8 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

Chapter 4

Cortical Memory

The cortical memory appears to compute in time steps of about 200 millisec-onds. During this time the memory settles to a state or decision that is usedto guide the next action. The decision may be thought of as a branch in aprogram. The action may be overt as in reaching for a target, or covert as insimulating the reach internally. States of the memory are coded hierarchicallywith high level neurons coding abstract features and low-level neurons codingconcrete features of the sensory surround.

4.1 Cortical Memory is Different

One of the most difficult things to come to grips with is that the basicmachinery of the brain seems to work in a way that is nothing like ourconscious perception. Take a moment and look around you taking special careto tune in to the seemlessness of conscious experience. The your perceptionof the world seems clear, unaffected by the inhomogeneous resolution of theretina which provides clear resolution only inside a tiny degree of visualangle. Colors seem to be everywhere in your visual field even though thecolor measuring cones are concentrated need the center of the visual field.And the world seems stable and continuous even though it is sampled verydiscontinuously via discrete gaze points.

How could this continuous perception happen? One constraint that op-erates is that the experience of conscious perception is slow compared tothe brains mechanisms. Just generating a sentence takes several secondswhether audibly for external consumption or internal as part of our own

1

Page 2: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

2 CHAPTER 4. CORTICAL MEMORY

train of thought. Given the 300 millisecond unit of the memory, conscioussentence generation is slower by a factor of ten. Now movies in the cinemaare discrete taking 16 frames a second. But we perceive them as continu-ous because the brain’s basic perceptual machinery cannot keep up with therapid frame changes. We would certainly be upset with a 3 frame per secondmovie however! Why are we not bothered by the slow sampling of the eyes?It could be that the way our brain works is that it is agenda driven. Whenwe think, we interrogate the environment rather than passively absorbing it.The interrogation process starts with programs that use conscious temporalscales but these in turn use the much faster perceptual programs. These infact are so fast that we perceive their operations as continuous. Wheneverwe frame a question about our sensory surround, it is answered by the per-ceptual machinery instantaneously (from the perspective of the time scaleused by conscious experience). Thus the world appears continuous.

The reason for discussing conscious perception right before looking at thecortex as memory is that the two are not so directly related. Just jumpinginto facts about cortical memory without this prelude can lead to confusion,because inevitably one tries to relate conscious experience directly to some-thing called ”memory.” If fact they are distantly related, in that the cortex ispart of the complex of all the forebrain’s subsystems that create the consciousexperience of memory. Later when we develop the learning of programs thatuse cortical memory as a component, it will become easier to relate corticalmemory to our everyday ideas about memory.

4.2 Table lookup strategies

The cortical memory has a job to do in its instant. During a 300 millisec-ond fixation it may have to read an emotional expression, size up a scenefor danger or see whether a door is ajar. These operations must be doneextraordinarily fast. Consider that each neuron in the cortex typically com-municates at the rate of about 10 spikes peer second. This means that in 300milliseconds, there is only time to send 3 spikes. From this cursory analysisits obvious that the answer has to be already in the memory and looked up.For this reason we say that the cortical memory uses a table lookup strategy.

It is useful to compare the cortical table lookup strategy to one you arealready familiar with: That of looking up a telephone number in a telephonebook (No fair using whitepages.com or equivalent). In the phone book all the

Page 3: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.3. HOW TO BUILD A CAM 3

numbers are precomputed and indexed by surnames. Since you are smartyou will use a divide and conquer strategy. You’ll divide the book into twoparts, determine which of the two your surname is in and repeat the processon that portion of the book. Formally the number of operations you’ll needis proportional to the logarithm of the number of names. With a 100,000entry book you’ll need about ten tries to get to the right page. Plus you hadbetter have the surname handy. You will be at a loss if you only have thestreet address.

The cortex as a lookup strategy differs from the phone book in two ways.First of all it has to be even faster that your divide and conquer strategy.It has to take a few tries independently of the size of the entries. Formallywe call this constant time. The other improvement the cortex has is that itwill work for any index or fragment of an index. So notionally if you only nopart of the street address and something about the surname it may be ableto complete the entry for you. This property is termed content addressablememory or CAM. Figure 4.1 is a very oversimplified version of what the cortexcan do, and incorrect since the cortex does not store pictures in any literalsense, but it is still an important simulation as it illustrates the dynamicsinvolved. When pictures are stored in this memory they can be retrievedjust by specifying some of the correct values. Once this is done, connectionsin the memory neurons are able to quickly complete the pattern of storedvalues.

Content addressable memory is a wonderful concept. If you have a storedpattern you can retrieve it by specifying some of the parts of the pattern.The wonderful part is that the exact parts do not matter, you just have toget reasonably close to your pattern. You smell a fruit pie and it reminds youof your grandmother’s home. The idea would be that the who complex pie-grandmother-home pattern was stored as a unit and can be thus recoveredby specifying some arbitrary bits and pieces.

4.3 How to build a CAM

Content Addressable Memory is such an important concept that we willenflesh the mechanics behind the example. To do this, we will use an ab-straction of a neuron as representing a memory bit. This abstraction needssome additional structure to set the value of the bit. The model of a neuronthat is widely used for this purpose is shown in Figure 4.5. As you know from

Page 4: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4 CHAPTER 4. CORTICAL MEMORY

Figure 4.1: The fundamental idea behind a Content Addressible Memoryillustrated with three examples that evolve in time horizontally. Each imageis composed conceptually of a large array of neurons that are either firing(+1or white) or off(-1 or black). Each neuron is connected to all the others withthe formula described below. The initial condition causes some neurons tofire, and others to remain off. This in turn causes the neurons to change stateat the subsequent time step. Owing to the way the synapse strengths havebeen picked, the final patterns that were coded via synapses are completed.

Page 5: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.3. HOW TO BUILD A CAM 5

xi

xj

wij

Figure 4.2: The abstraction of a neuron used for the simple CAM modelrepresents the state of each neuron i as a firing rate xi and the synapseconnecting neurons i and j as a real numbered weight wij.

Chapter 3, the model neuron is much simpler than a real neuron, but we’lluse the term neuron anyway for simplicity. A large number of interconnectedunits is a network . The state of a unit will be discrete, either 1 or −1. (Inlater chapters an analog state of a real number between 0 and 1 will be veryimportant, but here the ±1 state is the easiest to work with; it will simplifythe analysis.) Real neurons can represent negative numbers in at least twodifferent ways. One is to have a baseline firing rate mean “zero,” but themore common way, which we have already introduced, is to use two neuronsto represent the signal, one for the positive part and one for the negativepart.

The key parameters that define the memories are the weights that modelsynaptic connections between units, as shown in Figure 4.5. Biologicalsynapses are very complicated, so modeling them with a single real num-ber is a gross simplification. The goal is to specify the weights so that thechanges in the state vector can function as a memory. The weight wij betweenunit j and unit i will be a signed real number.

Another helpful thing to do is to have a notation to refer to the entirecollection of neurons’ firing patterns at once. As you know, we can do thisby describing the state of the network in terms of a state vector x. Thisjust has the states of individual units as components; that is, for n units,x = (x1, x2, x3, . . . , xN). All we have done is designate a single symbol tocover the entire state of the network. This coding scheme is illustrated inFigure 4.1 where each panel is composed of a collection of N neurons thatare either OFF (-1, black) or ON (+1, white). As time evolves, the panelsin Figure 4.1 has a different state vector for each of the time steps.

Page 6: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

6 CHAPTER 4. CORTICAL MEMORY

1

- 1

g(x)

x

Figure 4.3: For mathematical convenience, the activation function used forthe model neurons is extremely simple. It has output +1 if the combineinput is positive and an output of -1 if the combined input is negative.

The next step is to define a way of changing the state vector. The firstequation one might first think of using is to make the state dependent on theproduct of the input and synaptic weights as we have done before:

xi(t + 1) = wi · x(t) (4.1)

The problem with this equation is that xi in this case could be arbitrarilylarge. To restrict it to be either ±1, let us use a limiting function:

g(u) =

{+1 u ≥ 0−1 otherwise

This function is simply the sign of its argument and is shown in Figure 4.3.Thus Equation 4.1 becomes

xi(t + 1) = g (wi · x(t))

For the CAM we will use Hopfield networks in which every unit is con-nected to all the others[2] and furthermore, the weights connecting the unitsare symmetric; that is, wij = wji. It turns out that, for any given problem,it is relatively easy to pick wij so that the network will act like a memory.Given a set of P patterns to be stored xp, p = 1, . . . , P , the appropriatesetting for the weights is given by

wij =P∑

p=1

xpi x

pj (4.2)

This is a form of a Hebbian Rule which makes the weight strength propor-tional to the product of the firing rates of the two interconnected units. This

Page 7: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.3. HOW TO BUILD A CAM 7

rule is named after Donald Hebb, who first suggested that synaptic strengthsmight be determined by the correlation between pre- and postsynaptic neuralfiring patterns[1].

Example As an example of the weight calculation, pick the following threepatterns to be stored in a memory:

In this case P = 3 and letx1 = (−1, 1, 1,−1, . . .)T

x2 = (1, 1,−1,−1, . . .)T

x3 = (−1, 1,−1, 1, . . .)T

Now calculate one of the weights, for example, w23:

w23 = x12x

13 + x2

2x23 + x3

2x33

= (1× 1) + (1×−1) + (1×−1) = −1

Once the weights have been picked, the memory is ready for operation.The input layer is initialized to some state. Next the output is computed.This is then used as the input at the next time step.

Is this going to work? Suppose that you had just one pattern and considerconnection w23 as before. That connection’s value is now just x2x3. So if westarted with the pattern, it would have its product w23x3 equal to x2x3x3.But this is just x2 because x3x3 is always 1 no matter whether the value ofx3 is plus or minus. So the effect of the pattern is to deposit the sum of Ncopies of x2 as ‘raw’ input. And when the function g is finished with it, it willof course be whatever value x2 was in the first place. All this is a longwindedway of saying that if the pattern starts out with each neuron in the correctstate, it will stay put.

What about the case when there is noise in the pattern? Well since noiseis random, then sometimes they will accidently have the correct value andsometimes the opposite. For a large value of N the accidentally correct valueswill cancel the accidentally incorrect values and the correct input will stillresult in the pattern being correct, usually after one step. You can now seewhat other patterns might do. They will have the effect of raising the chancethat a particular neuron will have the incorrect value. Mathematically itcan be shown, although we will not do it here, that as long as the number ofpatterns is small compared to N , even though the original pattern has lots of

Page 8: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

8 CHAPTER 4. CORTICAL MEMORY

mistakes, as the examples in Figure 4.1 do, they will converge to the correctpattern as the examples do. Physicists have a nice way of describing this.Remember that we can represent the entire image as a point, as we do withthe a particular pattern vector x0. Now imagine all the possible x s. Therewill be ones that are close enough to x0 so that when the equations are used,they will each gravitate to the stored pattern. The area that contains thesex0 is called the ‘basin of attraction’ for x0.

Figure 4.4: The abstract picture of content addressable memory depictsstored patterns as points where each point stands for an entire pattern.Nearby patterns are also points, but the neural dynamics is such that theywill gravitate to a stored point. The region defining the points that will movetowards a particular point is referred to as that point’s basin of attraction.The figure shows two such basins, but there will many others, one for eachpattern.

The basin of attraction concept is a useful abstraction but it is veryimportant to draw the right lessons from it. For the cortex, a pattern willnot be an image. If it were then some mechanism would have to look at it andwe would be right back at the retina where we started! Each pattern will besome coding of a state. The state of course might refer to some aspects of animage, but in general it will have lots of other information - perhaps the foodvalue of something in the image if you are hungry or a logical assessment ofthe prospects of catching it - at this point we cannot say exactly what mightbe represented. Later on there will be some suggestions. The importantpoint about the CAM is its neural dynamics that can quickly compute a

Page 9: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.4. A MAP AND ITS PROPERTIES 9

state from a vast number of nearby starting points, This has to be done veryquickly as it is the platform from which the brain can then compute actions.

4.4 A Map and its Properties

In chapter three you saw how the information in a small image patch couldbe coded. Owing to the need to save spikes, it seems to be important to thebrain to sink its effort into coding the useful patterns that come up in theworld, and these are a very small fraction of the possible patterns owing tothe structure of the world. Taking this strategy, the first set of features thatappear mathematically are small linear discontinuities such as those shownin Fig 3.9. This result represented the coding for a small image patch. Whathappens for the entire visual field? It turns out that the first area of the cortexthat receives the connections from the LGN, V1 or striate cortex, is laid outin a retinotopic map. This means that as one traverses V1 the responses ofcells are sensitive to the structure in a traverse of the visual field. Figure 4.1Ashows the results of Tootell’s classic experiment. A monkey stares a screenwith a flickering radial pattern shown in the top segment. The monkey isanesthetized so that the eyes are fixed and centered on the target. At thesame time the monkey receives an injection of 2-deoxyglucose - think sugar-that is rapidly taken up by spiking neurons. The monkey is sacrificed and itsV1 is put on a photographic plate. The bottom segment shows the developedimage which reveals bands of cells that sent large amounts of spikes. To gainan idea of the scale, it is estimated that there are 150,000 cells per cubicmillimeter, so one is looking at a lot of cells here.

An inspection of the two patterns reveals another important feature ofvision and that is that the central area of the pattern is exaggerated in corticalarea. This of course reflects the fact that the central area of the visual fieldis sampled in the retinas by their high resolution foveas. Thus, for vision,the brain allocates neural hardware roughly per image sample. It has beenestimated that if we had brains that handled foveal resolution over the wholevisual field instead of the central one degree, the brain would have to weigh300 pounds. You can see why its cheaper just to move the eyes!

Now its time to reconcile two sets of information. First the propertiesvary across retinotopic space and second for each local area there are a col-lection of cells coding for different orientations. It turns out that there isa compromise between grouping the cells together based on orientation and

Page 10: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

10 CHAPTER 4. CORTICAL MEMORY

A

B

C

Table 4.1: (left) (right)[Permission pending]

based on position. This compromise is shown in Figure 4.1B where at a muchsmaller scale the cells of similar orientation are color coded. So for a smallspatial area all the orientations are represented. Orientation is just one ofa handful of properties that are represented. The short list includes oculardominance (which eye has the strongest connection), color, and direction se-lectivity (which direction of motion is the cell sensitive to). Just by countingyou can quickly see that there is a limit to the number of properties that canbe represented because if there are too many there would not be enough cellsto represent all the combinations. Figure 4.1C shows the smallest scale whereReid has used a calcium imaging technique to show which cells are responsiveto different directions of motion. It turns out that in the rat the differentdirections are all mixed up, but in the cat (shown here) the directions in alocal spatial area are segregated as shown in the figure.

4.5 Hierarchical Maps

In the Figure 4.1 the different neurons representing the image all have thesame status. However nothing could be further from the way the cortical

Page 11: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.5. HIERARCHICAL MAPS 11

neurons are organized and so it is important to refine the description toinclude the notion of abstraction. Think for a moment about the visualmeasurement of movement. How can you detect that an object in an imageis in motion? The easiest way is to detect changes in the individual imageelements, termed pixels. Since pixel values changing over time can signalmotion. The difficulty is that pixel changes do not always signal motion butcan also signal illumination changes, like the lights turning on and off. Itturns out that image motion can be separated from illumination changes butonly after a certain amount of computation.

The first step in that process is the computation of optic flow. Optic flowis in the form of an image: At every point in the visual field there is thevelocity of that point in space but we only see part of the three dimensionalmotion. The component along the ray of projection is lost. Nonetheless thecollection of remaining velocities form an image. In the cortex, midway upthe hierarchy, the collection of neurons that are sampling this image form aretinotopic map. What this means is that if we could stimulate the retinaat some specific location with a small patch of motion, then there wouldbe a corresponding small group of cells in the optic flow map that wouldrespond. For this reason the pure motion image, termed optic flow, is anabstraction of the photometric change image. This process can be repeated.If you think of the optic flow, that can represent many different kinds ofmotion from all the motion going in one direction to all the motion elementssignaling a random direction. But now think of the optic flow fields thatyou might experience as you walk through the world. These are very special.For example if you look straight ahead and move straight ahead, all the flowvectors will point radially out of the center of your image. This is given vividdemonstration in fiction when the Star Wars spaceships jump to light speed.The particular patterns in the flow images that result from self motion areuseful in navigation, but again are extracted with a computational cost.

The elaboration of the different forms of representation that are com-putable from the movement in an image is an abstraction hierarchy. Just toemphasize this point, in summary the hierarchy is:

SELF MOTION|

OPTIC FLOW|

TEMPORAL CHANGE

Page 12: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

12 CHAPTER 4. CORTICAL MEMORY

In the same way color has a similar hierarchy. Look around you. Nodoubt you can see a vivid impression of the colors of things. And you canname the colors of the things you see. Now suppose that you went and got agreen lightbulb and illuminated your scene( hopefully you are indoors now!)with the new bulb. You would still be able to name the colors correctlyand not only that you would perceive them to be their normal colors. Thisis a remarkable achievement because now most of the light coming fromyour object is green! The unraveling of the contributions of the differentialreflection of the light spectrum by the surface from the spectrum of theillumination can be done but requires - computation. once the color of thesurfaces of the object have been identified, they can serve as a label forthat object. As Michael Swain showed, this can be particularly effective formulticolored objects. But to agglomerate the amount of different colors of amulticolored object requires some computational bookkeeping. So the upshotis another hierarchy:

COLOR OBJECT LABELS|

SURFACE REFLECTANCE|

COLOR IMAGE

The general principal is that all of the cortex is organized into hierarchalmaps; the two examples from vision that we described, color and motion,are the kinds of computational chains that are repeated for different sensorymodalities such as audition and touch. One way of remembering this orga-nization is to think of an old roll-down window shade construction. Suchshades were on spring-loaded rollers and could be pulled down to shield thesun. In this analogy, the hippocampus is the roller and the cortex is theshade material itself. The hippocampus organization is very close to a rolledup sheet so it is very cooperative in this analogy! At any rate neurons nearthe roller represent very abstract concepts. The closer one moves to thelower edge the more we approach the sensory-motor periphery and the moreconcrete the cell codes become.

Like computer graphics, the cortex distinguishes between the transfor-mations that describe where objects are (they may be in motion) from theirproperties such as their color and texture, or what they are. Massively so asthere are two huge streams of two-way connections between maps. One forthe WHERE computations that goes over the top or dorsal/parietal part of

Page 13: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.5. HIERARCHICAL MAPS 13

Figure 4.5: Illustration of the first two visual maps in monkey visual cortex.Top The top view of a cortex stained with the tracer cytochrome oxidase,which is taken up by neurons that spike frequently. the stain clearly indicatesthe border between areas V1 and V2. Bottom left A schematic of the priciplethat shows how areas are related hiearchically. Layer II-III neurons in an arealower down in the abstraction hierarchy send their terminals to the upperarea’s layer IV. Going the other way, neurons in II-III and V-VI connect tothe corresponding layers in the lower area. Bottom right These connectionscan be revealed by anatomical staining [Permission pending].

Page 14: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

14 CHAPTER 4. CORTICAL MEMORY

Figure 4.6: A portion of the visual maps in monkey cortex illustrating theWHAT-WHERE distinction. The calculation of motion and self motion isdone in the V1-V2-MT-MST hierarchy and the calculation of color propertiesis done in the V1-V2-V4 hierarchy[Permission pending]

the cortex and another for the WHAT computations that goes along the sideor temporal part of the cortex. Figurefig:Hierarchy summaries some of thevisual maps, again for a monkey, showing the WHAT-WHERE dichotomy.

4.6 What does the cortex represent?

While we know the kinds of things that are represented in the cortical mem-ory, we are far from knowing exactly what gets represented. One generalcomputational way of thinking of what is being represented is in terms ofvery complex codes. Paul Revere’s revolutionary war audience just had toknow how the opposing British forces were going to attack. Hence the famous

Page 15: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.6. WHAT DOES THE CORTEX REPRESENT? 15

light code ”one if by land, two if by sea.” In the same way we have to respondto every day circumstances, by triaging them quickly. Is it safe to make aright turn now? You have to analyze a complex traffic pattern and come upwith a go/no go decision. Detailed coding models have been difficult to comeup with for the most abstract maps, but we saw that some success has beenachieved at the earliest map, V1. Olshausen and Field have shown that thegroups of cells there can be though of as trying to code image patches withthe smallest number of firing neurons. This was the basis for Figure 3.9 inChapter 3. Those neuron’s receptive fields result because most of the timethe local photometric variations look like linear edges separating two areasand the number of neurons that could represent this was constrained to besmall.

The smallest number of neurons cannot be the whole story though. Con-sider Figure 4.7 which shows sets of colored lines. You should be able to havethe impression of a translucent blue disk on top of a set of black lines. Thisis an illusion because if you compare small white areas either side of the diskby covering everything else up, you will see that they are the same shadeof white. Debates rage over how this happens. At one extreme, researchersargue that the early retinotopic areas of cortex fill in. that is make the neu-rons that are in charge of blue in empty space here actually send more spikesand have developed extensive models to show that this is possible [?]. At theother extreme, researchers claim that the ‘blue disk’ neural code does nothave to be literal. Just turning on the abstract representation, whatever itis, is enough to experience the blue disk image. From the CAM model wouldlead us to also allow for a middle ground. The neurons that are receivingunambiguous input try an complete a pattern. That pattern will involveboth abstract and concrete parts.

There is evidence that the lower cortical areas are modulated. In a re-lated experimental setup, von der Heydt has shown that neuron firing ratesare sensitive to the figure that edges belong to. If an edge is part of a completefigure that ‘owns’ it, it will fire more than if it is part of the background.What this means is that, to work this out, the neuron has to be commu-nicating with other neurons in the map. The fact that the firing measurerepresents an important property of the world - closed boundaries - meansthat the computations are very sophisticated.

Thus the evidence shows that the cortex represents a coded version ofthe world, but also that that code can be adjusted to reflect not only thelow level pixel correlations needed to produce edge-like cells in V1, but also

Page 16: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

16 CHAPTER 4. CORTICAL MEMORY

Figure 4.7: An illustration of the idea of pattern completion. There is notranslucent blue disk! Seeing it is an illusion created by having all the bluelines just the right lengths so as to from the boundary of a disk.[Permissionpending]

complex figural properties. Thus the correlation that the CAM is sensitiveto can be very high order.

Finally we will introduce another important idea and that is that of adistributed representation. For that we turn to Tanifuji’s studies of objectcoding in monkey cortex. The representation of abstract object properties isdone at the end of the temporal (WHAT) neural hierarchy. Since the portionof the cortex is relatively flat there in a rhesus monkley it can be studiedwith a special optical technique. When neurons fire the use oxygen and so re-move it from blood. relatively deoxygenated blood is darker than oxygenatedblood and so can be measured with an optical microscope backed by imagedifferencing techniques. The result is that such areas can be detected anddisplayed on top of the conventional image as is done in Figure 4.9.

To summarize, the cortex is a complex Cam indeed, as it does a numberof things simultaneously:

1. It must compactly code sensed data. Most of what we know abouthow this is doen comes from work with vision and image coding. InV1 a model that measures the statistics of images in the natural worldproduces similar receptive fields to those observed experimentally. As

Page 17: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.6. WHAT DOES THE CORTEX REPRESENT? 17

A

B C

Figure 4.8: (A) Reversible figure depends on the ‘boundary ownership.’ Ifthe boundary belongs to black you should see two monkey faces, otherwisea vase. (B) Recordings in V2 show that edge cells are sensitive to boundaryownership constraints. (C) Spike histograms reveal that this ownership ques-tion is resolved extremely quickly, only 25 milliseconds after stimulus onset.[Permission pending]

the main feature of this model is adding a cost to spikes, there is hopethat this might be a general principle.

2. The interpretation of images that we see reflects complex ways in whichthe world can present itself. Thus an image might reflect a translucentobject that is occluding another object further in the background. Il-lusions can be created that play on the brain’s ability to compute thesekinds of interpretations. Such illusions suggest sophisticated neuralcodes and these have been detected experimentally.

3. At the most abstract sites in the cortex, the coding of object prop-

Page 18: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

18 CHAPTER 4. CORTICAL MEMORY

Figure 4.9: A view of a portion of temporal cortex that represents objectproperties. The figure shows the results of two kinds of experiments. Oneis the a technique that uses optics to measure small blood flow opacitiesproduced by firing neurons (a). The other is the familiar single spike record-ing method ( b and c).The final panel (d) shows that they are correlated.The important point of the figure is that the represenation of objects isdistributed across this cortical map with a single object causing modulatedfiring in disparate parts of the map.[Permission pending]

Page 19: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

4.6. WHAT DOES THE CORTEX REPRESENT? 19

erties is distributed. With properties being represented by neurons isdisparate areas of cortical maps. This may reflect the fact that wetypically only need to access the properties of one or a small numberof objects at a time. Thus the coding of objects can be shared withoutproducing confusion as to which one is being referred to.

All these aspects are very important in understanding the cortex, yetperhaps the most important one is still missing. This is the use of the in-formation in a program. Recall the example of the previous chapter wheremonkeys learned to discriminate motion patterns. While the cortical areascan have all the properties just discussed they still need to be capable of be-ing configured to solve particular problems. How does the monkey’s cortex‘know’ that a particular motion sensitive neuron is the one to pay atten-tion to in making a decision? In the monkey we can reach for the answerof training. The monkey learns the task arduously over many months. Soperhaps there is time to ‘burn in’ a circuit that does exactly what is neededby adjusting synapses. However humans can do tasks like this with only thebarest of instructions. So there is yet a final thing to explain and that is howthe cortex can be configured to test the representations that it has in thecorse of computing a program’s crucial state. This aspect is so importantthat it will be discussed in its own chapter.

Page 20: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

20 CHAPTER 4. CORTICAL MEMORY

Page 21: Chapter 4 Cortical Memory - University of Texas at Austindana/Ch4.pdf · CORTICAL MEMORY train of thought. Given the 300 millisecond unit of the memory, conscious ... The cortex as

Bibliography

[1] D. O. Hebb. The Organization of Behavior. Wiley - New York, 1949.

[2] J. J. Hopfield and D. W. Tank. Computing with neural circuits: a model.Science, 1986.

21