the uses of colour vision: behavioural and physiological distinctiveness of colour stimuli

Published online 30 July 2002

The uses of colour vision: behavioural andphysiological distinctiveness of colour stimuli

Andrew M. Derrington*, Amanda Parker, Nick E. Barraclough,Alexander Easton, G. R. Goodson, Kris S. Parker, Chris J. Tinsley

and Ben S. WebbSchool of Psychology, University Park, Nottingham NG7 2RD, UK

Colour and greyscale (black and white) pictures look different to us, but it is not clear whether thedifference in appearance is a consequence of the way our visual system uses colour signals or a by-productof our experience. In principle, colour images are qualitatively different from greyscale images becausethey make it possible to use different processing strategies. Colour signals provide important cues forsegmenting the image into areas that represent different objects and for linking together areas that rep-resent the same object. If this property of colour signals is exploited in visual processing we would expectcolour stimuli to look different, as a class, from greyscale stimuli. We would also expect that adding coloursignals to greyscale signals should change the way that those signals are processed. We have investigatedthese questions in behavioural and in physiological experiments.

We find that male marmosets (all of which are dichromats) rapidly learn to distinguish between colourand greyscale copies of the same images. The discrimination transfers to new image pairs, to new coloursand to image pairs in which the colour and greyscale images are spatially different.

We find that, in a proportion of neurons recorded in the marmoset visual cortex, colour-shifts inopposite directions produce similar enhancements of the response to a luminance stimulus.

We conclude that colour is, both behaviourally and physiologically, a distinctive property of images.

Keywords: colour vision; image segmentation; object detection; figure-ground; monkey; vision

1. DISTINCTIVENESS OF COLOUR STIMULI

(a) BackgroundThis paper is concerned with the fact that we readily dis-tinguish between stimuli that contain colour variationsand those that do not. Quite apart from any significancethat we might attach to particular colours, or any use wemay choose to make of colour information, we distinguishbetween the presence and the absence of colour in a scene,a pattern or a picture.

This distinctiveness of colour might be something thatwe learn from our experience of colour and greyscaleimages in photography, in the cinema and on television.However, this paper makes the case that the distinc-tiveness could arise naturally from the visual consequencesof the presence of colour in an image. We shall argue insupport of this latter position in three ways. First, we shallargue that the contribution of colour and, in particular,hue variations to image segmentation makes colour imagesqualitatively different from greyscale images. Second, weshall show behavioural evidence that a monkey, the com-mon marmoset Callithrix jacchus, rapidly learns to dis-criminate between colour and greyscale images. Finally,we shall show physiological results indicating that the

* Author for correspondence ([email protected]).

One contribution of 14 to a Discussion Meeting Issue ‘The physiologyof cognitive processes’.

Phil. Trans. R. Soc. Lond. B (2002) 357, 975–985 975 2002 The Royal SocietyDOI 10.1098/rstb.2002.1116

presence of colour modulates the responses of somecortical neurons to a luminance stimulus. Crucially, in thislast instance, the nature of the modulation depends onlyon the presence of colour; it is independent of the actualcolour that is present.

In the following two subsections, we consider the roleof colour, and particularly of hue information, in theidentification of stimuli and in the segmentation of images.

(i) Hue and stimulus identificationSpecific hues, or hue combinations, may be used to

detect and identify specific objects or classes of object. Forexample, many edible fruits may be detected against abackground of foliage by the fact that they are somewhatredder than the foliage. This type of information, in whichspecific hues indicate specific types of object, is obviouslyimportant. Indeed, there is evidence to support the pro-posal that in old-world monkeys the photoreceptor pig-ments that support red–green colour discriminations haveevolved to maximize the sensitivity of the discriminationbetween fruit and foliage (Osorio & Vorobyev 1996;Regan et al. 1998; Sumner & Mollon 2000). However,there is no reason to suppose that links between specifichues and specific objects would lead to a situation wherecolour images, as a class, look different from greyscaleimages. Each hue might have its own significance, but theimportance of a coloured image would depend on whichhues it contained, not on the mere presence of the vari-ations in hue that differentiate colour from greyscale

976 A. M. Derrington and others Uses of colour vision

images. We consider a more general kind of informationprovided by hue variations that could make their merepresence in an image important.

(ii) Hue and image segmentationOne of the fundamental problems of vision is image seg-

mentation—deciding which bits of the image should begrouped together because they represent parts of the sameobject. Indeed, the segregation of objects from each otherand from the background is a prerequisite for most of thecognitive operations that can be performed on visual stim-uli. Edge detection, the detection of lines in the imagealong which the luminance changes abruptly, has alwaysbeen regarded as an important early computation in imagesegmentation (Marr 1976, 1982; Marr & Hildreth 1980).However, although edges may indicate the boundariesbetween objects of different reflectance, luminance edgescan also be misleading because they can be caused byshadows. This problem can be confounded when objectsocclude parts of each other so that a single object may bedivided into several distinct areas. Hue provides two typesof information that can resolve these two kinds of con-fusion.

First, retinal images, like most images received from thenatural world are formed by light reflected from a groupof objects illuminated by a single light source. It is a physi-cal fact that, in such images hue boundaries cannot becaused by shadows. Hue boundaries must representboundaries between differently reflecting materials, andthus potentially between different objects. Consequently,algorithms for segmenting images can be driven by search-ing the image for boundaries between areas of differenthue (Hurlbert 1989).

Second, even when they are in different levels ofshadow, if the different parts of a partially occluded objectare made of the same material and lit by the same lightsource, they will have the same hue, though not necessar-ily the same lightness. Thus, in principle, the differentparts of an image that represent the same object can belinked together by visual processing, which links togetherareas of the same hue (Mollon 1989).

Figure 1 shows these aspects of colour in a diagram thatrepresents the colour information available to the animalthat we used in our behavioural experiments, the malemarmoset. Male marmosets are dichromats, so the infor-mation available to them about any single colour can berepresented in a two-dimensional plot showing the relativeextent to which it excites both photoreceptor systems(Tovee et al. 1992; Travis et al. 1988). Thus, each colourcan be represented as a point in this space. We are inter-ested in colour boundaries, such as that between an objectand its background. The convention that we adopt is torepresent each boundary as an arrow with its tail on thepoint representing the background colour and its head onthe point representing the foreground colour.

The thickly drawn arrows in figure 1 all point eithertowards or away from the origin. They represent colourchanges that do not change the ratio of excitations of thetwo photoreceptors. They are changes of intensity withoutany change in hue: each arrow represents a boundary thatcould be caused by a shadow. In fact, the heavy arrows infigure 1 are all running along the same line pointingtowards the origin, and thus all represent different inten-

Phil. Trans. R. Soc. Lond. B (2002)

2.0

1.5

1.0

0.5

0

2.01.51.00.50

S-cone excitation

L-c

on

e ex

cita

tio

nFigure 1. Conventions for representing colour andluminance boundaries in images. Each point in the diagramrepresents the information about a hypothetical colourprovided by the photoreceptors of a dichromat, such as themale marmoset: the excitations produced in the twophotoreceptor systems, the L cones and the S cones. The L-cone excitation is plotted against the S-cone excitation.Borders between two areas of different colour within animage are represented as an arrow from one colour (usuallythe background colour) to the other (usually the foregroundcolour). The heavy arrows indicate boundaries that involve achange in lightness but no change of hue; they could becaused by shadows. The paler arrows indicate changes ofhue (and often also lightness). In an image of reflectingobjects lit by a single source, changes of hue indicatechanges in the reflecting material.

sities of the same hue. Their different start- and endpointscould all represent the same object seen in different levelsof shadow. The fact that they lie on the same line radiatingfrom the origin could be used to link together differentparts of the same object.

Different hues are encoded as different ratios of photo-receptor activation and would lie on lines radiating outfrom the origin in different directions. Shadows on objectsof each hue would be represented by arrows pointing upor down the hue line, towards the origin or away from it.The thinly drawn arrows in figure 1 represent colour-shiftsthat could not be caused by shadows. The clue to this isthat they point across the diagram along lines that do notpass through the origin. They represent boundaries in theimage that are associated with changes in hue and thusnecessarily correspond to changes in reflectance.

(b) Testable predictionsThese two potential uses of hue in image segmentation

mean that all images that contain hue variations form adistinct class for a visual system. Regardless of the actualhues involved, all images that contain more than one huehave the common property that colour algorithms can beused to segment them into patches corresponding to dif-ferently reflecting surfaces, and to link together the

Uses of colour vision A. M. Derrington and others 977

Figure 2. Experimental procedure: photograph of amarmoset retrieving his reward after rotating the stimuluscube. See text for further details.

patches that correspond to the same surface in differentlevels of shadow. Thus, if colour information is used insegmentation, we can make two predictions.

First, colour and greyscale images should naturally formdifferent categories and it should be easy to teach animalsto respond differentially to them. Specifically, once trainedto respond differentially to colour and greyscale images,they should extend the differential response—withoutfurther training—to novel pairs of colour and greyscaleimages, including pairs in which the colour image containsonly hue variations not present in the training set. In § 2,the behavioural section of this paper, we confirm this pre-diction by showing that male marmosets trained to dis-tinguish between pairs of greyscale and colour imagestransfer the discrimination to novel images and to novelcolours.

Our second prediction is that, at some point in the vis-ual pathway, hue boundaries should modulate visualresponses in a distinctive way. The modulation should bedifferent from that produced in neurons whose responsesare used to identify hue, in which different hues producedifferent responses and complementary hues usually pro-duce opposite responses. Rather, what should happen isthat the presence of a hue boundary should modulate theresponse of the neuron in a way that does not depend onthe actual hues. The neuron should signal the presenceof the hue boundary rather than either the hues that areassociated with it, or its sign. In § 3, the physiological sec-tion of this paper, we demonstrate that a small numberof neurons in the striate and prestriate cortex have theirresponses modulated by hue boundaries in this way.

2. BEHAVIOURAL CLASSIFICATION OF COLOURAND GREYSCALE STIMULI

(a) Behavioural training and testing(i) General method

Male common marmosets (Callithrix jacchus) weretrained to discriminate between pairs of images in a vari-ant of the Wisconsin General Testing Apparatus (Dias etal. 1996a,b) that could be mounted on the side of theirhome cage (see figure 2). One image was the positive, orrewarded stimulus, S�, and the other, the unrewarded


stimulus, S�. Each image was printed on card andmounted behind a transparent cover on the front face ofa backless hollow plastic cube, which measured ca. 25 mmon each side. Each cube was mounted on a vertical spindleand could be rotated by the marmoset in order to retrievethe reward, a small piece of marshmallow, from the backof the cube containing the positive stimulus.

Before the start of each trial, the marmoset was signalledto move to the back of his cage, 0.6 m from the stimuluscubes. Two screens, one opaque and one transparent,were placed in front of the stimulus cubes, which werethen loaded with the stimuli and the reward for the nexttrial. To start the trial, the opaque screen was removed,whereupon the marmoset inspected the stimuli and sig-nalled his choice by moving towards one of the stimuluscubes. His choice was recorded and the transparent screenwas removed, allowing him access to his chosen stimuluscube. If he was correct, he was able to rotate the cube,retrieve the reward and consume it, before returning tothe back of the cage for the next trial.

Using this procedure, marmosets performed between 40and 100 trials per day while being fed a normal foodration. Feeding took place after testing each day.

(ii) StimuliStimuli were photographic-quality prints 23 mm square,

produced by an inkjet printer (Epson Stylus Photo 750).Training stimuli were clip-art bitmaps printed in colour orgreyscale. Testing stimuli were complex two-dimensionalshapes drawn by doodling with a Bezier line tool in adrawing package (see figure 4a–c) and printed as a solidshape of a single colour on a grey or coloured background.

The reflectance spectra of printed samples of the col-ours used in the test stimuli were measured with a Data-color SF600 reflectance spectrophotometer at 10 nmintervals between 360 and 700 nm, and interpolated at4 nm intervals. The emission spectra of the lamps in thebehavioural testing apparatus were measured at 4 nmintervals using a SpectraResearch PR650 spectroradi-ometer. The marmoset cone absorption spectra were cal-culated at 4 nm intervals using the polynomialapproximation from Stavenga et al. (1993). Products ofsets of emission, reflectance and absorption spectra wereused to calculate the coordinates of the test-stimulus col-ours in a dichromatic cone-excitation space for the mar-moset.

(iii) Pre-training and trainingPre-training was designed to teach the marmosets to sel-

ect the rewarded stimulus. Once they had habituated tothe apparatus, marmosets were trained to retrieve piecesof marshmallow from the stimulus cubes. Then they weretrained to rotate the stimulus cubes to retrieve marshmal-low. Finally, they were trained to move to the back of thecage at the start of a trial before beginning trials in whicha pair of coloured clip-art pictures were used as stimuliand one of the images was consistently rewarded.

Ten pairs of pictures, chosen so that they contained awide range of colours and image features, and so that nocolour or image feature consistently predicted reward,were used as discrimination tests for pre-training. Mon-keys were trained for 40 trials per day on each discrimi-nation problem until they chose the positive stimulus on


100

75

50

25

00 5 10 1 2 3 4

discrimination number day

erro

rs t

o c

rite

rio

n

100

75

50

25

0

% c

orr

ect

(a) (b)

Figure 3. (a) Number of errors made by four monkeys in learning a colour/greyscale image pair to a criterion of 10consecutive correct choices, plotted against the number of discriminations learned. For Giles (squares) and Piers (circles) thecolour image was rewarded; for Rupert (triangles) and Hugo (diamonds) the greyscale image was rewarded. After the sixthdiscrimination the monkeys were trained for 2 days on a list of the first five discriminations. (b) Performance of the same fourmonkeys learning a list of 10 novel colour/greyscale discriminations over four days. Each monkey was trained on the list for 40trials per day.

at least 36 of the 40 trials on three consecutive days.Throughout training and testing, the correct stimulus wasalternated from side to side using a Gellermann sequence.

Once pre-training with mixed stimuli had been com-pleted, the marmosets were divided into two groups andbegan learning to discriminate between colour and grey-scale versions of the same clip-art picture. For two of them(Giles and Piers), the colour picture was rewarded and forthe other two (Rupert and Hugo), the greyscale picturewas rewarded. Once established for a given monkey, theassociation between colour and reward was retainedthroughout his training. At this stage, we excluded anypictures containing red, so that we could use red picturesto test whether the colour discrimination transfers to novelcolours. Each colour/greyscale discrimination was learnedto a criterion of 10 consecutive correct responses.

Figure 3a shows the numbers of errors made by fourmonkeys learning to distinguish between colour and grey-scale versions of the same picture to a criterion of ten con-secutive correct responses. Performance on the first sixdiscriminations was extremely variable. In the best case,Rupert had an immediate preference for the second grey-scale image and so learned the correct response with noerrors. In the worst case, Giles made almost 90 errorswhile learning the fourth image pair. All four learned thefifth image with no more than one error, but the sixthimage produced variable performance.

At this point, they were retrained to ensure that theirperformance was 90% correct over 40 trials for the firstfive image pairs presented as a list, i.e. on each trial, anyof the five image pairs could be presented with the con-straint that no pair could be presented for the nth time


until all had been presented n � 1 times. Giles, Rupertand Piers reached 90% correct on the first day of trainingwith the list, but Hugo took 2 days.

After this confirmation that they remembered the firstfive discriminations that they had learned, they resumedtraining with individual colour/greyscale discriminations.The last four discriminations were learned very rapidly.There were 20 errors by the four monkeys, and the modalnumber of errors per discrimination was zero. This sug-gests that, by this stage, the monkeys had learned to relyon the presence of colour to enable them to choose thecorrect stimulus and did not need to learn each imagepair separately.

Figure 3b confirms this impression. It shows perform-ance on a novel list of 10 colour/greyscale discriminations,which were presented in 40 trials per day over four days.Even on the first day, performance is between 80 and 95%correct. It improves slightly over the four days and isbetween 90 and 95% correct on day 4. This consistentlyhigh level of performance with novel stimuli indicates that,like humans, marmosets reliably distinguish images thatcontain colour variations from those that contain onlyvariations in lightness: colour is a distinctive stimulusquality.

(iv) Necessary controlsIn order to establish that the monkey’s discrimination

performance is based on the property of colour, we needto exclude two possibilities. First, we need to exclude thepossibility that the marmosets are choosing between col-oured and grey-scale images because of luminosity differ-ences rather than colour differences. Although we have no


??

1

2

3

2.0

1.5

1.0

0.5

0

0 0.5 1.0 1.5 2.0

S-cone excitation

L-c

one

exci

tati

on

(a)

(c)

(d )

(b)


Figure 4. Stimuli for colour discrimination testing. (a)Training stimuli and test logic for testing discriminationswithout differentially rewarding the discriminanda. In 80%of the trials, the monkeys were trained for reward on the listof four colour/greyscale pairs. In 20% of the trials, a pair oftest stimuli stimuli was presented and both stimulus boxescontained a reward. The monkeys’ performance on test trialsinformed us about their stimulus choices in the absence ofdifferential reward. (b) Five of the 10 test stimuli used indemonstrating that the colour/greyscale discrimination is notbased on an apparent luminance or contrast difference. (c)Five of the 16 test stimuli used in demonstrating that thecolour discrimination transfers to a novel hue. (d ) Training-and test-stimulus colours plotted in cone-excitationcoordinates with arrows indicating the borders betweenbackground and figure colours. The three heavy arrows showthe figure/ground borders of the training stimuli. The lighter,numbered arrows show the figure/ground borders of the teststimuli. No. 1 is the blue test in (b). No. 2 is the red ‘novelhue’ test from (c), which was successfully discriminated byHugo, Rupert and Piers but not by Giles. No. 3 is thelighter red that was successfully discriminated by Giles. Notethat the heavy arrows representing the training stimuli allrun across the diagram in a direction of increasing relative S-cone excitation (down and right), whereas the arrowsrepresenting the novel hue stimuli run up and left in adirection of increasing L-cone excitation.

a priori reason to suppose that there are luminance differ-ences between colour and greyscale stimuli, we do notknow the marmoset’s spectral luminosity function and itis possible that the colours that we used might appear tobe systematically brighter (or darker) than the greys withwhich they are matched. Second, we need to exclude thepossibility that the marmosets have simply learned a listof colours.

(v) Test procedureIn order to measure how the training transferred to

specific sets of test image pairs without differentiallyrewarding the responses to test image pairs and withoutlosing the effect of training, we used the following pro-cedure.

The animal was first trained on a list of fourcolour/greyscale image pairs to a criterion of 90% correctin the 40 trials of a day’s testing. Then, during testingsessions, 80% of the trials were ‘training’ trials and woulduse a stimulus pair from the trained list, and the rewardwould only be received if the correct image was chosen(colour or greyscale in accordance with the training historyof the particular monkey). Twenty per cent of trials, selec-ted at random, were ‘test’ trials. On test trials, a teststimulus pair was presented and both of the stimuluscubes contained a reward. Consequently, the monkey’sresponses on test trials are determined exclusively by histraining and not by reward contingencies on previoustest trials.

(b) Control for luminosity differencesTo eliminate the possibility that the marmosets use

luminosity differences rather than colour differences, weprepared a series of colour images that consisted of differ-ent shapes, all of the same blue on the same grey back-ground (see figure 4b). The comparison greyscale images


Table 1. Performance of the four monkeys on the different transfer tests.

test Giles (%) Hugo (%) Rupert (%) Piers (%)

luminance transfer 97.5 92.5 97.5 100transfer to novel hue (dark red) 41 95 100 98transfer to novel hue (light red) 97 — — —dissimilar image pairs 80 80 92.5 85

used the same grey background but, in 50% of them, theshapes were lighter than the background and in the other50% darker than the background. Test pairs of imagesdrawn at random from this list were used for every fifthtrial while the marmosets were performing discriminationsfor reward using a list of four greyscale and colour stimulithat they had previously learned (see figure 3a). Wheneverthe luminosity test pairs were presented, both choiceboxes were loaded so that whichever choice they madewould be rewarded. This ‘double baiting’ enables us todiscover how the monkey’s training controls his choicefrom each test pair without training them on the test pair(Walsh et al. 1993).

Table 1 shows the performance of each monkey on thetransfer tests. The overwhelming majority of their choiceswere consistent with the colour discrimination (Giles97.5%, Hugo 92.5%, Piers 100% and Rupert 97.5% onthe luminance transfer test). Clearly, their colour choicesare not based on luminosity differences between coloursand greys because otherwise they would have made incor-rect choices consistently, either on the light grey or on thedark grey images, depending on whether the blue that wehad chosen appeared brighter or darker than the back-ground.

(c) Transfer to novel colours and novel imagepairs

We used a similar procedure to rule out the possibilitythat the marmosets are simply using a catalogue of col-ours, compiled during the experiment, to guide theirchoices. To do this, we made a set of 16 test image pairsin which the coloured images all used shapes made froma deep red on a grey background (see figure 4c). None ofthe previously used coloured images had contained anyred. As before, test image pairs selected at random werepresented on every fifth trial while the marmosets wereperforming discriminations for reward using a pre-learnedlist of four greyscale and colour stimuli. Both of the imagesof the colour test pair were rewarded. Hugo, Piers andRupert transferred their colour discrimination to the novelred images: they scored 95%, 98% and 100%, respect-ively. Giles, however, did not transfer his colour discrimi-nation to red images: his mean score was 41%.

One possible reason for Giles’s failure is that he wasunable to determine that the red test pattern was colouredbecause he could not distinguish between the red test col-our and a dark grey. To investigate this possibility, wetried to train him to distinguish the red from dark grey byrewarding him only if he made the correct choice on trialswith red test patterns. His mean performance in 160 trialsover four days was 54% correct. Clearly, he was unableto make the discrimination, although his performancereached 58% on the fourth day, suggesting that he might


eventually have learned to do so. However, there can beno doubt that the reason why he did not transfer the col-our discriminations to red images is that, during the timehe was training with these, he was unable to distinguishthe red from a dark grey. It is possible that Giles’s failureto make this discrimination arises because he has a differ-ent colour vision phenotype from the others, but it is equ-ally probable that it is a difficult discrimination for all ofthem. We have since learned, with other stimuli, that Gilesis slower than the other three to learn difficult visual dis-criminations.

Having established that Giles’s failure to generalize thecolour/greyscale discrimination to red images was causedby his difficulty in discriminating that the stimulus wascoloured, we decided to test him again using a lighter ver-sion of the same hue, which should be easier to discrimi-nate. We changed the dark red to a lighter red (see figure4d) and repeated the transfer test. He transferred almostperfectly: his average score was 97% correct over 8 days.Thus, despite Giles’s early failure, all the monkeys trans-ferred the colour/greyscale discrimination to one or otherof the novel hues.

Finally, we used the same transfer procedure to test allthe monkeys to see whether the colour/greyscale discrimi-nation would generalize to a series of 10 greyscale/colourimage pairs in which the greyscale and colour images weredifferent spatial patterns. Although the transfer was notperfect, all of the monkeys scored at substantially abovechance (Giles 32 out of 40, Hugo 32 out of 40, Piers 34out of 40, Rupert 37 out of 40). The difference betweenthe lowest of these scores and chance performance isextremely significant (p � 0.005; �2-test).

These results show that, once they have solved a num-ber of discriminations in which colour is the common fac-tor, male marmosets rapidly and efficiently utilize colourto choose between new stimuli, even when those stimulidiffer in other respects. They classify novel image pairsaccording to whether they are colour or greyscale, evenwhen those images contain only hues that are not presentin the training set. They even categorize pairs of picturesthat differ in other respects, according to the colour/greyscale criterion.

The fact that an abstract image attribute, such as col-our, is used in such a versatile and consistent way to guidethe choice between novel stimuli is consistent with thenotion that colour is a highly conspicuous stimulus pro-perty. The rapid transition from needing to learn each dis-crimination by reinforcement training to immediate,almost errorless performance with new discriminations(shown in figure 3a) suggests that the conspicuousness ofcolour is not simply a consequence of the reinforcementcontingencies learned in our experiments, but is innate.

The fact that the colour/greyscale discrimination trans-


fers to novel colours on which the marmoset has not beentrained (shown in table 1) is consistent with the notionthat, like us, the marmoset includes all colour images ina common category and it is the selection of this categorythat has been reinforced. The most likely reason for theperceptual salience of stimuli that contain hue variationsis the usefulness of hue variations in image segmentation.This leads to the prediction that monochrome pictures,which are coloured but contain only a single hue, shouldbe classified as ‘non-coloured’ images. We have not testedthis prediction because it is not possible to present col-oured monochrome pictures on a coloured monochromebackground in our present apparatus. We are at presentbuilding apparatus that will make it possible to presentsuch stimuli in order to test this prediction.

The next question to consider is whether the behav-ioural distinctiveness of colour for the marmoset isreflected in the neurophysiology of cortical visual pro-cessing.

3. MODULATION OF PHYSIOLOGICAL RESPONSESBY COLOUR-SHIFTS

In considering what kind of physiological propertiesmight support the behavioural discrimination between thepresence and absence of colour, it is, first, important todistinguish it from normal colour discriminations. In nor-mal colour discriminations, the aim is to identify the huerather than merely to detect the presence or absence ofvariations in hue. Neurons that support hue identificationmust respond differentially to varying hue. For example,they may only respond to a narrow range of hues (Zeki1983a,b) or they may respond in an opponent fashion tocomplementary hues (Wiesel & Hubel 1966). Neither ofthese types of response will support discriminationbetween the presence and absence of colour. An essentialrequirement is a response that signals the presence of huevariations in the stimulus but that does not differentiatebetween hues.

A second requirement arises if we assume that the func-tional significance of the ability to distinguish between thepresence and the absence of colour is likely to be relatedto the use of hue in image segmentation. Hue boundariesallow us to identify a set of edges in the image that rep-resent reflectance changes, and hence could not be causedby shadows. In general, these hue boundaries will coincidewith luminance edges. Under this assumption, the appro-priate neural signal would be a change in the responseto a luminance stimulus when it is associated with a hueboundary, rather than an overt response to the hue bound-ary.

In order to look for neural signals that fulfil both ofthese requirements, we have measured the responses ofneurons in marmoset visual cortex to small patches ofmoving sinusoidal grating. The grating was of optimal spa-tial frequency and orientation (i.e. the spatial frequencyand orientation that produced the largest response), andthe patch was of optimal length and width. The gratingwas presented under three different conditions (see figure5). In the ‘control’ condition, the hue and mean lumin-ance of the patch of grating was the same as the surround-ing screen. In the two ‘test’ conditions, the patch waschanged in hue either in a bluish or in a yellowish direc-


2.0

1.5

1.0

0.5

0

2.01.51.00.50

S-cone excitation

L-c

one

exci

tati

on

Figure 5. Stimuli for the physiological experiment, plotted incone-excitation coordinates. The point in the centre of theplot represents the baseline colour and luminance of thedisplay monitor. To its right and left are the pointsrepresenting the colour backgrounds. The heavy arrowsrepresent the range of colours covered by each of thesinusoidal gratings. The light arrows represent, for eachbackground, the range of colour changes between the screensurrounding the grating patch (which remained at thebaseline colour and luminance of the monitor) and thedifferent parts of the grating. The different symbols representthe same information plotted for L-cones of the threedifferent possible peak wavelengths. Circles represent563 nm, inverted triangles 556 nm and upright triangles545 nm.

tion while the surrounding screen was maintained at thesame hue and luminance. Thus, at the boundaries of thegrating patch, the screen changed both in luminance andin hue.

(a) Physiological recordingPhysiological preparation for recording from neurons

with microelectrodes and for presenting visual stimuliwere essentially as have been described previously(Felisberti & Derrington 2001).

(i) Visual stimulationVisual stimuli—spots or patches of different colours and

luminances—and patches of sinusoidal grating werepresented on a Sony Multiscan GDM200PDST monitorat a frame rate of 120 Hz by a Macintosh computer run-ning custom-written software. Stimuli were adjusted inposition and size and, if appropriate, in orientation andspatial frequency, in order to maximize the response ofthe neuron. Multiple stimuli, patches of different sizes,colours and contrasts, which could be combined withsinusoidal gratings of different contrasts, were randomlyinterleaved so that responses to stimuli that were beingcompared were accumulated concurrently. Stimulus lumi-nances were set using a lookup table to compensate forthe nonlinear relation between the luminance and appliedvoltage of the display. For coloured stimuli, cone exci-


tations were calculated using measurements of the emis-sion spectrum of the display screen, made at 4 nmintervals in the range 380–780 nm. Cone spectra were cal-culated using the approximations of Stavenga et al. (1993)for peak wavelengths of 423 nm for the short-wavelength-sensitive cone, and 545, 556 and 563 nm for the differentpossible long-wavelength-sensitive cones.

The colour changes of the grating patch are illustratedin cone-excitation coordinates in figure 5. They were cal-culated to produce a large variation in the short-wavelength cone signal and minimal variation in the longwavelength cone signal. Such colour changes produce astrong response in colour-opponent neurons driven byblue cones. However, in this case we are not interested indirect responses to the changes in hue of the grating patch.Instead, we are interested in the possibility that the changein hue will modulate the response to luminance.

Figure 5 shows the hue boundaries associated with thegratings on patches of different hues. The heavy double-headed arrows connect the colours corresponding to thepeaks and troughs of the sinusoidal grating. When thepatch of grating is the same hue as its surround, the arrowsare aligned along a line pointing to the origin. Both ofthe changes in hue of the grating patch change the lateralposition of the arrow, without changing its orientation, sothat it is no longer aligned with the origin. Thus, a mech-anism designed to ‘mark’ spatial modulations of colourthat include hue would mark the luminance changeswithin these gratings when they are presented on the col-oured patches.

The thin arrows in figure 5 show the colour changesthat occur at the edge of the patch of sinusoidal grating.They connect the point that represents the luminance andcolour of the unmodulated display with the points thatrepresent the luminances and colours of the peaks, thetroughs and the zero crossings of the sinusoidal gratingson coloured patches. These colour changes also includehue shifts—none of the arrows lies on a line that passesthrough the origin. Thus, both at their edges and withinthe patches of sinusoidal grating, there are strong hueshifts.

(b) Response of a single neuron to colour andluminance modulations

Figure 6 shows how the hue signals modulate theresponses of a striate cortical neuron to a sinusoidal grat-ing. Histograms a–c show the response to the moving grat-ing presented alone (a) and superimposed on the twocoloured patches (b,c). When the grating is presented ona coloured patch, the response is bigger by ca. 20%. Thetwo complementary coloured patches cause approximatelyequal enhancements of the response magnitude. However,when the patch colours are presented in isolation(histograms e and f ), they elicit no response.

The coloured patches are spatially and temporally uni-form: the colour-shift is turned on at the start of the stimu-lus presentation and off at the end. This raises thequestion of whether the lack of response in histograms eand f is because the neuron is insensitive to the colours ofthe patches, or because it is insensitive to their spatial andtemporal frequencies.

Histograms g and h answer the question. They show theresponses of the same neuron to colour modulations at


200

100

0

200

100

0

200

100

0

200

100

0

200

100

0

200

100

0

200

100

0

200

100

00 0.2 0.5 0 0.2 0.5

seconds seconds

spik

es s

_1

(a) (e)

(b) ( f )

(c) (g)

(d ) (h)

Figure 6. Responses of a neuron in striate cortex to thedifferent stimuli illustrated in figure 5. (a) Response to asinusoidal grating of optimal spatial frequency, length, widthand orientation, moving at 4 cycles s�1, presented on abackground the same hue and luminance as the rest of thescreen. (b) Response to the same sinusoidal grating as (a)presented on a background shifted towards the blue. Thearea surrounding the grating remained at constant hue andluminance. The background shift occurred abruptly andcoincided with the onset of the grating. The response isbigger than in (a). (c) Response to the same gratingpresented on a background shifted away from the blue(towards yellow). The response is bigger than in (a).(d) Response to the blank screen. (e) Response to thebackground shift towards blue, but with no grating.( f ) Response to the background shift away from blue,presented with no grating. (g) Response to the sinusoidaloscillation of the background between its two extremecolours (towards blue and away from blue) at 4 Hz.(h) Response to a sinusoidal grating of optimal spatialfrequency, length, width and orientation, which modulatesthe screen between the two extreme colours of thebackground and moves at 4 Hz in the same direction as thegrating used in figure 6a–c.

optimal temporal frequency of either a spatially uniformpatch (g) or a grating of optimal spatial frequency, orien-tation and size (h). Both when it is spatially modulatedat the optimal spatial frequency and when it is spatiallyunmodulated, the chromatic modulation (in which thecolour oscillates sinusoidally between the colours of thebluish and yellowish patches) elicits no discernibleresponse, whereas the response to a luminance modu-lation of the same temporal and spatial frequency is about100 spikes per second (figure 6a). Thus, the colour-shift


0.5

0

–0.5–0.5 0 0.5

enhancement by yellow background

enh

ance

men

t b

y b

lue

bac

kg

rou

nd

Figure 7. Plots of the extent to which the two backgroundsenhanced the responses of neurons recorded in the striatecortex, area 18 and the LGN. The enhancement is measuredas the log of the magnitude of the response in the presenceof the background divided by the magnitude of the responsein the absence of any background (i.e. the log of the ratio ofthe two response magnitudes). The enhancement by the bluebackground is plotted against the enhancement by the yellowbackground. Cells recorded in the striate cortex are plottedas circles, those recorded in the prestriate cortex are plottedas squares, and those recorded in the LGN are plotted astriangles. For each LGN cell the response magnitude ismeasured as the magnitude of the Fourier component in thedischarge histogram that corresponded to the temporalfrequency of the grating. For each cortical cell, the responseis measured either as the mean firing rate during thepresentation of the grating, or as the magnitude of thecorresponding Fourier component, whichever is the larger.

enhances the response to luminance without generatingan explicit response of its own.

(c) Summary of colour modulation effectsThe non-selective enhancement of neuronal responses

illustrated in figure 6 could potentially provide a basis forthe neuron to contribute to the discrimination betweenluminance-only edges and colour edges. In figure 7, wesummarize the responses of all the neurons that we havetested with the same set of stimuli by plotting the degreeto which the response is enhanced by the blue hue shift,against the degree to which it is enhanced by the yellowhue shift. Neurons whose responses are enhanced by bothhue shifts appear in the top right-hand quadrant and neu-rons whose responses are suppressed by both hue shiftsplot in the lower left-hand quadrant. Colour-selective neu-rons, in which the hue shifts have effects of opposite sign,appear in the other two quadrants. Although the majorityof the cortical neurons show no effect, a significant num-ber—three out of the six neurons recorded in area 18 (V2)and three out of the 18 neurons recorded in striate cortex(V1)—show an enhancement of their responses by bothhue shifts. One of the neurons recorded in area 18 shows asubstantial suppression of its response by both hue shifts.


Also shown in figure 7 are the responses of a larger sam-ple of lateral geniculate neurons in the same experiment.None of the LGN cells shows non-selective enhancementof responsiveness by the hue shift. Most of them lie closeto the origin, showing that the response to luminance isnot modulated by colour. A small number lie in the lowerright quadrant, showing that the two hue shifts haveopposite modulatory effects—this is the behaviour to beexpected of colour-opponent neurons.

4. ENHANCEMENT OF NEURAL RESPONSES AS ABASIS FOR BEHAVIOURAL DISTINCTIVENESS

OF COLOUR

(a) Striate and prestriate cortexWhatever the function of the perceptual salience of col-

our, it seems likely that its physiological basis is related tothe way that it enhances the responses of neurons earlyin the cortical visual pathway. The increases in responsemagnitude caused by the hue shift are clearly detectablein the averaged responses, but modest in size, amountingto no more than about a 30% increase in firing rate in theneurons that we have studied. In the one neuron wherewe observed a decrease in firing rate, the effect was moredramatic: both hue shifts reduced the response by morethan 50%. However, we shall focus on the functional sig-nificance of the increases in firing, since they are morecommon. The two questions to consider are, first, whetherthese physiological effects are a possible basis for thebehavioural results, and second, whether they are the mostlikely basis for such an effect.

There can be little doubt, in principle, that the physio-logical response facilitation that we have observed couldform the basis for the behavioural discrimination betweencolour and greyscale images. Although the increases inresponse magnitude are probably too small to be detectedin a single neuron, on a single presentation of a stimulus,the fact that they can be detected in the averagedresponses of individual neurons makes it extremely likelythat they could be detected in a single trial by averagingacross the responses of many neurons. We know that suchenhancements are possible: the sensitivity of macaquemonkeys in colour discriminations exceeds the sensitivityof the individual neurons that relay colour signals throughthe monkey LGN (Derrington et al. 1984; Derrington1992). Thus, if we assume that the response enhance-ments occur across the neuronal population with approxi-mately the frequency that we have observed in our sample,these increases in response magnitude should be easilydetectable on a single trial by averaging across neurons.

Of course, only averaging across the population of neu-rons whose responses are enhanced by colour would notbe enough to distinguish colour images from greyscaleimages. The response enhancements caused by colourcould be confused with response enhancements caused byincreases in contrast. The results of the luminance transfertest show that the discrimination between colour and grey-scale images is independent of the contrast of the greyscaleimages. To resolve the confusion, the responses wouldhave to be compared with those of similar neurons whoseresponses were unaffected by colour. Successful discrimi-nation between colour and greyscale images would require


a comparison between the averaged responses of the twopopulations of neurons.

Consequently, although the neurons that we haverecorded could contribute to the discrimination, it is likelythat the discrimination itself occurs at a later stage of vis-ual processing, where the signals carried by neurons likethose that we have recorded can be pooled. This, ofcourse, raises the possibility that the response enhance-ments that we record could be a consequence of the colourgreyscale discrimination fed back from the later stage,rather than providing the basis for the discrimination thatis carried out there.

(b) Likely involvement of later stages ofprocessing

Another reason for supposing that the discriminationbetween colour and greyscale occurs at a relatively latestage in visual processing is the potential role of hueboundaries in image segmentation (Hurlbert 1989). Theenhancement of physiological responses associated withhue boundaries that we have observed could be a by-product of the image segmentation process, which is likelyto depend on quite late stages of visual processing. Onesuggestion is that visual processing in cortical area V4 isrelated, in a general way, to the segmentation of surfacesin the image (Lennie 1998) and that this general segmen-tation includes aspects of selectivity to colour that aresensitive to differences in hue between the visual stimulusand its surroundings (Zeki 1983b). Lesions in V4 appearto disrupt pattern discriminations and complex object-related colour discriminations, but leave simple colour dis-criminations unaffected (Heywood & Cowey 1987; Hey-wood et al. 1992; Walsh et al. 1993). Thus, it is probablethat cortical area V4 is central to the physiological andbehavioural phenomena described in this paper. It isimportant to discover both how the responses of neuronsin V4 are modulated by hue shifts such as those used inthis study, and whether neurons in V1, V2 and V3 stillshow modulation of their responses by hue shifts when V4is removed or inactivated.

To answer this question, we need to record from striateand early prestriate cortex in a preparation, with highercortical areas inactivated or removed. It would also beinformative to carry out the behavioural task in a prep-aration with higher cortical areas removed. In this respect,it is significant that macaque monkeys with inferotemporalcortex lesions show subtle deficits in colour discrimination(Huxlin et al. 2000), although macaques have not, to ourknowledge, been tested on their ability to generalize acolour/greyscale discrimination.

5. THE USES OF COLOUR VISION

We propose that the utility of colour signals for imagesegmentation makes colour stimuli distinctive. If so, weshould also expect that the availability of colour signalswould improve our visual abilities. The evidence on thisquestion is sparse and suggests that colour improves per-formance but not speed. When monkeys or humans haveto classify images according to whether they contain ani-mals or food, colour makes no difference to the speed oraccuracy with which they can do the task (Fabre-Thorpeet al. 1998; Delorme et al. 1999, 2000). On the positive


side, the recognition of natural scenes is improved by theavailability of colour signals both during encoding andduring recall. The advantage of colour during encoding isinterpreted as a sensory effect, and may reflect its useful-ness in image segmentation (Gegenfurtner & Rieger2000).

(a) Different uses of L–M and short-wave systemsFor convenience, we have concentrated our experi-

ments on the short-wavelength colour-opponent channelby using male marmosets (which have only this channel)for the behavioural work and by targeting our physiologi-cal colour stimuli on the short-wavelength channel. In thefollowing paragraphs, we consider whether our con-clusions would also apply to the channel carrying signalsfrom the long- and medium-wavelength cones, the L–Mchannel.

Another reason for concentrating our experiments onthe short-wavelength opponent channel is that this is theprimordial colour vision system possessed by most mam-mals that have colour vision (Jacobs 1993). Its functionis therefore likely to reflect the primordial role of colourvision. It is clear that the short-wavelength sensitiveopponent channel contributes to human memory because,in human X-chromosome-linked dichromats, which lackthe L–M (red–green) colour-opponent channel, colour hasthe same enhancement of memory as it does in normaltrichromats (Gegenfurtner et al. 1998; Gegenfurtner &Rieger 2000). However, we do not know whether the L–M colour-opponent channel makes such a contribution.This raises the question of whether we would expect thecolour images that selectively stimulate the L–M colour-opponent channel of old-world primates to show the samedistinctiveness that we have observed in our experiments.

One reason for supposing that the L–M colour channelmight not be used in the same way as the short-wavelengthchannel is that its evolution is more recent (Mollon 1989)and has been linked to the specific colour discriminationsassociated with fruit eating (Osorio & Vorobyev 1996;Regan et al. 1998; Sumner & Mollon 2000). Conse-quently, we might expect signals in the L–M channel tobe used for identifying specific classes of object but notfor image segmentation. We have begun experiments withmacaque monkeys to test whether the two colour-opponent channels contribute in the same way to the dis-tinctiveness of colour images.

As in most other new-world primate species that havebeen studied, marmosets have a range of colour-visionphenotypes that can be accounted for by variations in thepigment sensitive to longer wavelengths, which are codedby a single gene on the X-chromosome. There are threesuch pigments, with peak sensitivities at 545, 556 or 563nm (Hunt et al. 1993). All marmosets share a commonshort-wavelength-sensitive pigment; males have a singlelong-wavelength-sensitive pigment and females have two,drawn at random from the three. Thus, all male mar-mosets and one third of female marmosets are dichromats,with only a single post-receptoral colour-opponent chan-nel (Tovee 1994). Because of the wide variety of colourvision phenotypes and the uncertainty about the extent towhich the marmoset visual system is wired to exploit theavailability of trichromatic signals in the minority of mar-mosets that possess them, we did not attempt to investi-


gate any contribution of a possible L–M channel either tophysiological responses or to behaviour. However, carewas taken to design our stimuli so that the identity of thelong-wavelength cone pigment that contributes to theshort-wavelength-sensitive colour-opponent channel isirrelevant. Figure 5 shows that the identity of the long-wavelength-sensitive cone makes only a tiny, quantitativedifference to the effect of the colour modulations and sovariations in cone type would not affect our results.

6. CONCLUSION

Both in their behavioural responses and in the responsesof neurons in areas 17 and 18, marmosets distinguishbetween spatial patterns that contain only variations inluminance and those in which hue also varies. We proposethat these distinctions are related to the use of colour sig-nals for image segmentation.

This work was supported by grants from the Wellcome Trustand the BBSRC, a Wellcome Prize studentship to B.S.W. andan MRC studentship to G.R.G. The authors thank StephenWestland and Sophie Wuerger for their help with colour cali-brations, Jackie Francis and Mel Williams for their help withbehavioural testing, and Peter Lennie and Anya Hurlbert forthe use of their software.

REFERENCES

Delorme, A., Richard, G. & Fabre-Thorpe, M. 1999 Rapidprocessing of complex natural scenes: a role for the magno-cellular visual pathways? Neurocomputing 26, 663–670.

Delorme, A., Richard, G. & Fabre-Thorpe, M. 2000 Ultra-rapid categorisation of natural scenes does not rely on colourcues: a study in monkeys and humans. Vis. Res. 40, 2187–2200.

Derrington, A. M. 1992 Relative sensitivity to colour and lum-inance gratings in human observers and monkey parvocellu-lar neurones. J. Physiol. 452, 286.

Derrington, A. M., Krauskopf, J. & Lennie, P. 1984 Chro-matic mechanisms in lateral geniculate nucleus of macaque.J. Physiol. 357, 241–265.

Dias, R., Robbins, T. W. & Roberts, A. C. 1996a Dissociationin prefrontal cortex of affective and attentional shifts. Nature380, 69–72.

Dias, R., Robbins, T. W. & Roberts, A. C. 1996b Primate ana-logue of the Wisconsin Card Sort Test: effects of excitotoxiclesions of the prefrontal cortex in the marmoset. Behav.Neurosci. 110, 872–886.

Fabre-Thorpe, M., Richard, G. & Thorpe, S. J. 1998 Rapidcategorization of natural images by rhesus monkeys. Neuro-Report 9, 303–308.

Felisberti, F. & Derrington, A. M. 2001 Long-range interac-tions in the lateral geniculate nucleus of the new-world mon-key, Callithrix jacchus. Vis. Neurosci. 18, 209–218.

Gegenfurtner, K. R. & Rieger, J. 2000 Sensory and cognitivecontributions of color to the recognition of natural scenes.Curr. Biol. 10, 805–808.

Gegenfurtner, K. R., Wichmann, F. A. & Sharpe, L. T. 1998The contribution of color to visual memory in X-chromo-some-linked dichromats. Vis. Res. 38, 1041–1045.

Heywood, C. A. & Cowey, A. 1987 On the role of corticalarea V4 in the discrimination of hue and pattern in macaquemonkeys. J. Neurosci. 7, 2601–2617.

Heywood, C. A., Gadotti, A. & Cowey, A. 1992 Cortical area-


V4 and its role in the perception of color. J. Neurosci. 12,4056–4065.

Hunt, D. M., Williams, A. J., Bowmaker, J. K. & Mollon, J. D.1993 Structure and evolution of the polymorphic photopig-ment gene of the marmoset. Vis. Res. 33, 147–154.

Hurlbert, A. 1989 Colour algorithms for image segmentation.In Models of brain function (ed. R. M. J. Cotterill), pp. 89–113. Cambridge University Press.

Huxlin, K. R., Saunders, R. C., Marchionini, D., Pham,H. A. & Merigan, W. H. 2000 Perceptual deficits afterlesions of inferotemporal cortex in macaques. Cerebr. Cortex10, 671–683.

Jacobs, G. H. 1993 The distribution and nature of colourvision among the mammals. Biol. Rev. Camb. Phil. Soc. 68,413–471.

Lennie, P. 1998 Single units and visual cortical organization.Perception 27, 889–935.

Marr, D. 1976 The early processing of visual information. Phil.Trans. R. Soc. Lond. B 275, 483-524.

Marr, D. 1982 Vision: a computational investigation into thehuman representation and processing of visual information. NewYork: W. H. Freeman & Co.

Marr, D. & Hildreth, E. 1980 Theory of edge detection. Proc.R. Soc. Lond. B 207, 187–217.

Mollon, J. D. 1989 “Tho’ she kneel’d in that place where theygrew. . .” The uses and origins of primate colour vision. J.Exp. Biol. 146, 21–38.

Osorio, D. & Vorobyev, M. 1996 Colour vision as an adap-tation to frugivory in primates. Proc. R. Soc. Lond. B 263,593–599.

Regan, B. C., Julliot, C., Simmen, B., Vienot, F., Charles-Dominique, P. & Mollon, J. D. 1998 Frugivory and colourvision in Alouatta seniculus, a trichromatic platyrrhine mon-key. Vis. Res. 38, 3321–3327.

Stavenga, D. G., Smits, R. P. & Hoenders, B. J. 1993 Simpleexponential functions describing the absorbency bands ofvisual pigment spectra. Vis. Res 33, 1011–1017.

Sumner, P. & Mollon, J. D. 2000 Catarrhine photopigmentsare optimized for detecting targets against a foliage back-ground. J. Exp. Biol. 203, 1963–1986.

Tovee, M. J. 1994 The molecular genetics and evolution ofprimate colour vision. Trends Neurosci. 17, 30–37.

Tovee, M. J., Bowmaker, J. K. & Mollon, J. D. 1992 Therelationship between cone pigments and behavioural sensi-tivity in a new world monkey (Callithrix jacchus jacchus). Vis.Res. 32, 867–878.

Travis, D. S., Bowmaker, J. K. & Mollon, J. D. 1988 Polymor-phism of visual pigments in a callitrichid monkey. Vis. Res.28, 481–490.

Walsh, V., Carden, D., Butler, S. R. & Kulikowski, J. J. 1993The effects of V4 lesions on the visual abilities ofmacaques—hue discrimination and color constancy. Behav.Brain Res. 53, 51–62.

Wiesel, T. N. & Hubel, D. H. 1966 Spatial and chromaticinteractions in the lateral geniculate body of the rhesus mon-key. J. Neurophysiol. 29, 1115–1156.

Zeki, S. 1983a Color coding in the cerebral-cortex—the reac-tion of cells in monkey visual-cortex to wavelengths and col-ors. Neuroscience 9, 741–765.

Zeki, S. 1983b Color coding in the cerebral-cortex—theresponses of wavelength-selective and color-coded cells inmonkey visual-cortex to changes in wavelength composition.Neuroscience 9, 767–781.

GLOSSARY

LGN: lateral geniculate nucleus

the uses of colour vision: behavioural and physiological distinctiveness of colour stimuli

Documents