retrospective revaluation and its neural circuit in rats
TRANSCRIPT
R
R
Aa
b
a
ARRAA
KAPNcM
1
tbaeiordacciigt[[a
0d
Behavioural Brain Research 223 (2011) 262– 270
Contents lists available at ScienceDirect
Behavioural Brain Research
jo u r n al hom epa ge: www.elsev ier .com/ locate /bbr
esearch report
etrospective revaluation and its neural circuit in rats
urore San-Galli a,b, Alain R. Marchanda, Laurence Decortea, Georges Di Scalaa,∗
Institut de Neurosciences Cognitives et Intégratives d’Aquitaine UMR 5287, Université de Bordeaux, CNRS, Avenue des Facultés, 33405 Talence Cedex, FranceCentre de Recherches sur la Cognition Animale UMR 5169, Université Paul Sabatier, CNRS, 118 route de Narbonne, 31062 Toulouse Cedex 09, France
r t i c l e i n f o
rticle history:eceived 4 March 2011eceived in revised form 11 April 2011ccepted 16 April 2011vailable online 22 April 2011
eywords:ssociative learningrefrontal cortexucleus accumbens-Fos
a b s t r a c t
Contingency learning is essential for establishing predictive or causal judgements. Retrospective reval-uation captures essential aspects of the updating of this knowledge, according to new experience. In thepresent study, retrospective revaluation and its neural substrate was investigated in a rat conditionedmagazine approach. One element of a previously food-reinforced Tone-Light compound stimulus waseither further reinforced (inflation) or extinguished (extinction). These treatments affected the predic-tive value of the alternate stimulus (target), but only when the target was a weakly salient stimulus suchas a Light, and the inflation/extinction procedure concerned the more salient element, that is the Tone. Asthe predictive value of the Light was decreased in comparison with a relevant control group, this reval-uation was interpreted as backward blocking, and not unovershadowing. This observation challengesretrospective revaluation models focused on acquisition and prediction error detection, and is better
agazine approach accounted for by retrieval-based associative theories such as the comparator model (Miller and Matzel)[5]. Immunohistochemical detection of the Fos protein after the test phase revealed activation of theorbitofrontal and infralimbic cortices as well as nucleus accumbens core and shell, in rats that exhibitedretrospective revaluation. Our results suggest that rats integrate successive experiences at the retrievalstage of retrospective revaluation, and that prefronto–accumbal interactions are involved in this function.
. Introduction
Contingency judgement in humans is essential for predic-ion and causal reasoning. Potential predictors or causes ofehaviourally significant events must be evaluated against onenother and this knowledge must be updated according to newxperience. Retrospective Revaluation (RR) is such a process whichnvolves cue competition. It occurs when one element of a previ-usly reinforced compound stimulus is subsequently either furthereinforced (inflation) or extinguished (extinction). RR is then evi-enced as an opposing decrease or increase of response to thelternate element of the compound. These changes are respectivelyalled backward blocking and unovershadowing. RR is considered ahallenge for classical learning theories [1,2], as it involves changesn the predictive value of the alternate element which is not phys-cally present. To address this problem, modifications of the rulesoverning acquisition of associations have been proposed, such as
he modified SOP [3] and the revisited Rescorla and Wagner model4]. Other theories such as Miller and Matzel’s comparator theory5] attribute RR to processing at a later stage, during retrieval of thecquired associations.∗ Corresponding author. Tel.: +33 540003842, fax: +33 40008743.E-mail address: [email protected] (G. Di Scala).
166-4328/$ – see front matter © 2011 Elsevier B.V. All rights reserved.oi:10.1016/j.bbr.2011.04.024
© 2011 Elsevier B.V. All rights reserved.
RR has been primarily described in human subjects perform-ing causal judgements tasks such as the “allergist scenario” (e.g.[3,6,7]). Increasing evidence for RR in rats is accumulating [8–13],but it is still reputed an elusive effect [14], suggesting that theappropriate experimental conditions have yet to be identified.Recently, the relative salience of the compounded elements hasbeen suggested as an important factor for RR [12]. In a first exper-iment, we report evidence for backward blocking in an appetitivemagazine approach learning task, in conditions where the moresalient stimulus was manipulated.
The brain circuits of RR are still poorly known, and Experiment2 aimed at documenting this issue. In human subjects, two fMRIstudies revealed increased neuronal activity within the right dor-solateral prefrontal cortex and the nucleus accumbens during theinflation/extinction stage of an allergist scenario [15,16]. In rodents,to our knowledge, the neural correlates of RR have not been stud-ied so far. In order to identify the brain areas activated in RR,immunohistochemistry of the Fos protein was used. We focusedour analysis of c-Fos expression on the test phase of the experiment,so as to investigate the neural correlates of RR retrieval processes.Structures of interest comprised components of the prefrontal cor-
tex, namely cingular, prelimbic, infralimbic, orbitofrontal cortices,limbic areas such as nucleus accumbens, hippocampus, entorhinalcortex, amygdala and lateral septum, as well as sensory cortices(visual, auditory, and gustatory). The results provide evidence for
A. San-Galli et al. / Behavioural Brain Research 223 (2011) 262– 270 263
Table 1Experimental design of Experiment 1. In stage 1, a Tone-Light compound was rewarded by food (+) for all groups. In each group, one element was designated as target cueand the other as competing cue. In stage 2, the competing cue (Tone or Light) alone was presented. Inflation groups received rewarded exposures (+) to the competing cue,whereas Extinction groups received unrewarded exposures (−). Half the animals in each control group were presented with rewarded Clicker trials, and the other half withunrewarded Clicker trials. All groups were also exposed to a Darkness cue (not shown in table) on alternate trials, to control for the amount of food reward and contextconditioning. A reward was delivered on the Darkness cue only in groups receiving unrewarded Tone, Light or Clicker. In a third stage, the predictive status of the target cuewas assessed. Control groups were tested with either the Tone or the Light as target.
Groups Stage 1: Compound training Stage 2: Competing cue training Test: Target cue
Inflation (I) Tone +
LightExtinction (E) Tone-Light + Tone −Control (C) Clicker +/−Inflation (I) Light +Extinction (E) Tone-Light + Light −
as
2
2
v(tcita
2
(temaebcwoF
2
pTesaeowLdtoCCBictTLbtwao
Control (C)
ctivation of a corticostriatal circuit during retrieval of a revaluedtimulus.
. Materials and methods
.1. Subjects
Male Long-Evans rats were obtained from a commercial supplier (Elevage Jan-ier, France). They weighted 310–340 g at the onset of training for Experiment 1n = 52) and 310–400 g (n = 31) for Experiment 2, and were food-deprived and main-ained at 90% of their initial weight. Rats were housed in pairs on a 12:12 h light/darkycle, with all experiments being conducted during the light cycle. These stud-es were in agreement with the French Directive (87/148, permission 33 05075o GDS) and the European Community Council Directive (86/609/EEC) regardingnimal research.
.2. Apparatus and stimuli
Conditioning and testing took place in eight identical chambers40 × 35 × 30 cm, Imetronic, France) made of grey PVC on three sides withransparent Perspex on the opening face and a grid floor. Each chamber wasquipped with a dispenser delivering 45 mg food pellets (Bio-Serv) into a recessedagazine. A houselight (4 LEDs on the ceiling of the box) provided illumination
nd a ventilation fan generated a background noise of approximately 65 dB. Stimulimployed were a Tone (4000 Hz, 75 dB) and a flashing Light (2.5 cycles/s) producedy two LEDs mounted to the left and right of the magazine. In addition, a Clicker (6ycles/s) and a Darkness cue (houselight turned off) were used. Stimulus durationas 10 s. On rewarded trials, a single food pellet was delivered 2 s before the offset
f the stimulus. Cues were controlled by a computerized interface (Imetronic,rance) that also recorded magazine entries.
.3. Behavioural procedure
The behavioural procedure was adapted from [12]. Experiment 1 involved twoossible treatments (inflation, extinction) and a control. The design is depicted inable 1. During a first stage (compound training), both stimuli (Tone-Light) werestablished as predictors of food. This stage consisted of 18 daily compound trainingessions, each containing 12 rewarded Tone-Light compound presentations, withn average intertrial interval (ITI) of 5 min (range: 3 to 7 min 40 s). One of theselements (Tone or Light, counterbalanced) was designated as target cue and thether as competing cue. In a second stage (competing cue training, 8 days), the ratsere presented with only one element of the compound, either the Tone or the
ight so as to alter its predictive status. Two Inflation groups were submitted toaily inflation sessions consisting of 6 rewarded exposures to the Tone (n = 10) orhe Light (n = 7). Two Extinction groups received daily extinction sessions consistingf 6 unrewarded presentations of the Tone (n = 10) or the Light (n = 8). In each of twoontrol groups (n = 10 and n = 7), half the animals were presented with 6 rewardedlicker trials, whereas the other half received 6 unrewarded Clicker presentations.ecause the Clicker stimulus had not been presented before, no RR was expected
n these control groups. In addition, to control for the amount of food reward andontext conditioning, all groups were exposed to the Darkness cue on alternaterials, with a reward delivered on the Darkness cue in groups receiving unrewardedone, Light or Clicker presentations, but not in groups receiving rewarded Tone,ight or Clicker presentations. Average ITI was 2.5 min, with an average interval
etween rewarded trials of 5 min. The critical test occurred in a third stage wherehe predictive status of the other (target) cue was assessed. On the test day, all ratsere tested with 5 unrewarded presentations of the target cue which had beenbsent during stage 2 (ITI: 5 min). Control groups were tested with either the Toner the Light as target. Magazine entries were recorded before and during the stimuli.
ToneClicker +/−
Experiment 2 was performed on a different set of rats with only two groupsTone Inflation (n = 16) and Tone Extinction (n = 15) treated as described above.
2.4. Immunohistochemistry
Rats from Experiment 2 were sacrificed with a lethal dose of pentobarbital90 min after the beginning of the test. They were perfused transcardially with aNaCl 0.9% solution followed by paraformaldehyde (PFA) 4% in 0.1 M Phosphate Buffer(PB). The brains were removed, postfixed overnight in PFA 4% and transferred to a PB0.1 M/sucrose 30% solution for 48 h at 4 ◦C. Serial horizontal sections (50 �m thick)were cut on a freezing microtome and stored in PB 0.1 M. Free-floating sectionswere incubated for 12 h with a primary rabbit polyclonal anti-Fos antibody (SantaCruz Biotechnology, USA), diluted at 1/5000 in a blocking solution (BS: PB 0.1 M,BSA 0.1%, Triton 0.2%, goat serum 2%). Sections were then incubated with a biotiny-lated goat anti-rabbit antibody (1/2000 in BS, Jackson Immunoresearch, USA) for 2 h.Staining was revealed using the avidin-biotin peroxydase method (ABC kit, VectorLaboratories) coupled to diaminobenzidine as chromogen (DAB, Sigma-Aldrich).
2.5. c-Fos quantification
Quantitative analysis of c-Fos positive cells was performed for a subset of15 rats. Animals were selected on the basis of their performance at test (upperand lower half of the Extinction and Inflation groups, n = 8 and n = 7 respectively)thereby focusing on those that showed the RR phenomenon. These rats did not differfrom the others during compound and competing cue training stages. c-Fos countswere obtained with an Explora Nova imaging station, consisting of a computerizedimage-processing software (Mercator®) combined with an optical microscope (LeicaDM6000) and a camera (Opsonic Microfire). Structures of interest were definedaccording to the Paxinos and Watson atlas [28]. Immunoreactive cells were countedbilaterally for each entire structure on a single horizontal slice, except the prelimbiccortex and the hippocampus, for which two sections were used. The mean count permm2 in each structure and for each animal was square-root transformed to improvehomogeneity of variance.
2.6. Data analysis
Behavioural data (magazine entries) were collected during the CS, subtractedfrom an equivalent period prior the CS and converted per minute to generate aCS minus preCS measure. In stages 1 and 2, the reward delivery period (last 2 s ofeach stimulus presentation) was excluded from the analysis. For the test phase, theresponses were collected during the last 5 s of the CS as responses to the Light stimu-lus were found to have a slow onset. Data were analysed with the StatView® software(version 5.0.1) using repeated measures ANOVAs with between-subject factors pro-cedure (Inflation vs. Extinction) and stimulus (Light vs. Tone), complemented byStudent’s tests when appropriate. ANOVAs and t-tests were also conducted on c-Fosresults. The ̨ risk for rejection of the null hypothesis was fixed at 0.05.
3. Results
3.1. Retrospective revaluation in magazine approachconditioning
During compound training in stage 1, rats gradually acquiredmagazine approach, since magazine entries increased and reached
an asymptotic level (mean number: 17.96) within 12 sessions inall animals (effect of sessions: F(17,46) = 54.73, p < 0.001). Duringcompeting cue training (Fig. 1, left panels), the Inflation groupsmaintained their rate of magazine entries across days, whereas
264 A. San-Galli et al. / Behavioural Brain Research 223 (2011) 262– 270
Fig. 1. Retrospective revaluation in magazine approach conditioning. Left: Stage 2. Evolution of the responses (mean minus baseline and S.E.) to the Tone stimulus (A) or theL (E) ob rimeng
ts
(yeivfwFtttptaTs
daa
ight stimulus (B) in separate groups of rats submitted to inflation (I) or extinctionaseline and S.E.) to test presentations of the alternate stimulus for the various experoup, significantly differed from the control group.
he Extinction groups clearly extinguished their responses to thetimulus presented.
A two-way ANOVA confirms the effect of reward procedureinflation or extinction), F(1,31) = 12.87, p < 0.01. In addition, anal-sis of the linear effect of sessions demonstrates the differentialffect of inflation vs. extinction procedures, since there was a signif-cant effect of procedure, F(1,31) = 10.8, p < 0.01, and stimulus (Lights. Tone), F(1,31) = 5.9, p < 0.05, without interaction between theseactors (F(1,31) = 1.39, p > 0.1). The ANOVA also indicates that thereere significantly more responses to the Tone than to the Light,
(1,31) = 5.91, p < 0.05, irrespective of reward procedure (interac-ion F < 1). Moreover, a Student test performed for the first day ofhis stage reveals a significant difference between the two Extinc-ion groups (receiving non-reinforced Tone or Light), [t(16) = 2.46,
< 0.05]. Animals actually showed fewer responses to the Lighthan to the Tone, as the result of compound conditioning. This resultgrees with Liljeholm and Balleine’s [12] data, indicating that theone overshadowed the Light within the compound and that thealience of the Tone was much larger than that of the Light.
On the test day, rats were presented with the cue that was absenturing competing cue training. Performances of the various groupsre shown in Fig. 1 (right panels). Both the overall level of responsend the effect of procedure clearly depended upon the stimulus
f that stimulus over 8 sessions. Right: Test. Rate of magazine entries (mean minustal groups (C: Control group). (*) p < 0.05. The inflation group, but not the extinction
tested, F(1,46) = 38.50, p < 0.001 (interaction F(2,46) = 3.73, p < 0.05).Rats tested with the Light still showed weaker responses but pre-sented a significant effect of procedure, F(2,27) = 3.30, p = 0.05.Responses in the Inflation group were significantly lower than inthe Extinction group, t(18) = 2.57, p < 0.05, thereby demonstratingRR. Further analysis showed that only the Inflation group differedfrom the Control group, t(18) = 2.33, p < 0.05. This result indicatesthat the revaluation consisted of backward blocking (inflating theTone in stage 2 induced a decrease in responses to the Light), andnot unovershadowing (extinction of the Tone had no effect onresponses to the Light). No such effect was obtained for rats testedwith the Tone. These rats failed to show any effect of procedure,F(2,19) = 1, n.s., suggesting that manipulations of the Light stimulusas a competing cue did not induce revaluation.
3.2. Neurobiological circuit of retrospective revaluation
The second experiment aimed at identifying neuronal areasinvolved in RR, using the behavioural procedure established in
the previous experiment. According to the results of the previ-ous experiment, only manipulations of the Tone in stage 2 wereattempted. Animals were sacrificed following the test and theirbrains were processed for immunostaining. c-Fos expression was
A. San-Galli et al. / Behavioural Brain Research 223 (2011) 262– 270 265
BA Nucleus AccumbensPrefrontal areasIE
Sqrt
Fos
cou
nts/
mm
²
Cg PL IL OFC
*
*
0
4
8
12
16
20
24
28
32
Core Shell
*
**
0
2
4
6
8
10
12
14
16
*
0
5
15
10
Entr
ies
per m
in
I E
Behavioural test
Fig. 2. Revaluation-induced c-Fos expression in prefrontal and accumbens areas. Mean and S.E. (square root transformed) of the density of c-Fos-positive nuclei followingtest with the Light. I: Tone-Inflation group (black bars); E: Tone-Extinction group (white bars). Cg: anterior cingulate cortex; PL: prelimbic cortex; IL: infralimbic cortex; OFC:o l. Inset
eaicaapoewsaia[a
fiis
Fti
rbitofrontal cortex; Core: nucleus accumbens core; Shell: nucleus accumbens shelwo groups when tested to the Light stimulus. (*) p < 0.05; (**) p < 0.01.
valuated in sixteen areas, including prefrontal, temporal and stri-tal regions. As specific areas of the rat prefrontal cortex arenvolved in uncertainty and conflict resolution [17], adaptation toontingency changes [18,19], evaluation of outcomes values [20]nd inhibitory control [21–23], part of these structures might bectivated in our revaluation task. The hippocampal formation (hip-ocampus proper and entorhinal cortex) was considered, becausef its role in within-event learning by integrating events of differ-nt modalities into a unified representation [24,25]. The amygdalahich is widely accepted to mediate the encoding of affective or
ensory representation of biologically significant event [26] waslso selected. Nucleus accumbens was included as this structures related to the dopaminergic control of switching of attentionalnd behavioural resources to behaviourally important stimuli27]. Finally, control areas including sensory cortices were alsonalysed.
The behavioural results of both groups replicated those of therst experiment at all stages. Magazine entries during stage 1
ncreased and reached a similar asymptotic level; effect of ses-ions: F(17,493) = 22.64, p < 0.0001. During stage 2, the two groups
ig. 3. Infralimbic activation during retrospective revaluation. Microphotography of c-Fohe Extinction (B) group. Horizontal slices were taken at Bregma −4.60. Lower panels (C anflation procedure elicits more Fos activation than the extinction procedure. PL: prelimb
t: behavioural responses (average magazine entries minus baseline and S.E.) of the
differed in their responses to the Tone, as expected. The ANOVAconfirmed this effect of the group, F(1,29) = 11.66, p < 0.01, whichinteracted with the sessions, F(7,203) = 3.33, p < 0.01. At test (seeinset in Fig. 2), rats from the Inflation group showed significantlyfewer responses to the Light stimulus, relative to those from theExtinction group (t(29) = 2.33, p < 0.05), thereby confirming the RReffect.
Fig. 2 depicts the c-Fos immunostaining data of the subset ofrats (n = 7 for group Inflation and n = 8 for group Extinction). Aglobal ANOVA indicated a significant interaction between groupsand neural structures [F(15, 195) = 2.95; p < 0.001], allowing amore detailed analysis of these effects. In prefrontal areas, thenumber of Fos positive neurons was increased for group Infla-tion compared to group Extinction, in the infralimbic cortex,t(13) = 2.4, p < 0.05, as well as the orbitofrontal cortex, t(13) = 2.75,p < 0.05 (see Figs. 3 and 4). Similar differences were observed
in cingular and prelimbic cortices but did not reach signifi-cance [t(13) = 0.88, p > 0.05 and t(13) = 1.98, p = 0.07 respectively].In nucleus accumbens, an increased number of Fos-positive neu-rons was observed in group Inflation, both in the core and shells immunostaining in the medial prefrontal cortex of rats from the Inflation (A) andnd D) show rectangular areas of the infralimbic cortex at higher magnification. Theic cortex; IL: infralimbic cortex.
266 A. San-Galli et al. / Behavioural Brain Research 223 (2011) 262– 270
F of c-FE C andm
d(vfinl
4
R
FE(
ig. 4. Orbitofrontal activation during retrospective revaluation. Microphotographyxtinction (B) group. Horizontal slices were taken at Bregma −4.60. Lower panels (ore Fos activation than the extinction procedure.
ivisions, t(13) = 2.28, p < 0.05 and t(13) = 3.56, p < 0.01 respectivelysee Fig. 5). Examination of 10 other areas (Table 2), includingisual and auditory cortices and limbic areas, revealed no other dif-erence between the Inflation and the Extinction groups, therebyndicating that the observed effects in the prefrontal cortex anducleus accumbens were not due to background or unspecific
abelling.
. General discussion
The experiments presented in this study provide evidence forR in rats in a conditioned magazine approach procedure, and they
ig. 5. Accumbal activation during retrospective revaluation. Microphotography of c-Foxtinction (B) group. Horizontal slices were taken at Bregma −6.60. Lower panels (C and DSh) divisions. The inflation procedure elicits more Fos activation than the extinction proc
os immunostaining in the orbitofrontal cortex of rats from the Inflation (A) and the D) show rectangular areas at higher magnification. The inflation procedure elicits
further attempt to identify the underlying functional neural net-work. RR was found to be asymmetrical, as it was observed wheninflation/extinction (competing cue training, stage 2) concernedthe tone, that is the more salient element of the compound, butnot when it concerned the light, that is the less salient element.Expression of the immediate early gene c-Fos was used to iden-tify changes of activity within brain areas during the test phase.Increased expression of c-Fos was within the prefrontal cortex
(infralimbic and orbitofrontal cortices) and the nucleus accumbens(shell and core), suggesting that neural activation of a cortico-striatal circuit may support a comparison process occurring at theretrieval stage.s immunostaining in the nucleus accumbens of rats from the Inflation (A) and the) show rectangular areas at higher magnification of both the core (C) and the Shelledure.
A. San-Galli et al. / Behavioural Brain
Table 2Revaluation-induced c-Fos expression in sensory cortices and limbic areas Mean andS.E. (square root transformed) of the density of c-Fos-positive nuclei following testwith the Light in Tone-Inflation and Tone-Extinction groups. CA1, CA2, CA3: CornusAmmonis 1, 2 and 3; DG: dentate gyrus.
Brain area Inflation Extinction
Visual cortex 30.83 ± 1.28 26.86 ± 1.45Auditory cortex 23.31 ± 2.81 23.03 ± 1.48Gustatory cortex 12.63 ± 1.80 12.31 ± 2.15Lateral septum 5.96 ± 0.92 4.75 ± 0.73Hippocampus
CA1 9.38 ± 0.61 8.08 ± 1.02CA2 6.59 ± 0.44 7.12 ± 0.75CA3 6.51 ± 0.44 6.05 ± 0.58DG 6.88 ± 0.17 6.41 ± 0.78
iflaapdetsoi
meaatbe[mto
turgieiadLTrrwiagccb
Rpr
Entorhinal cortex 19.14 ± 1.19 17.93 ± 2.04Amygdala 7.44 ± 0.61 7.67 ± 0.99
Our results confirm and extend previous studies reporting RRn rats, using conditioned suppression [e.g. 11,13], conditionedavour-preference [29] and conditioned magazine approach [12],s in the present study. Evidence for RR remains sparse, andn opposite effect, i.e. mediation, has been described in otheraradigms such as conditioned taste aversion [30–32, unpublishedata]. Observation of RR rather than mediation may depend on sev-ral factors yet to be identified. Mediation observed in conditionedaste aversion may be attributed to the use of stimuli that do notufficiently differ in salience. A substantial difference in the saliencef the elements of the compound may actually favour competitiventeraction and be required for RR.
Our data demonstrate RR in the group tested with the Light ele-ent of the compound, but not in the group tested with the Tone
lement. The Tone element was far more salient than the Light,s indicated by the differential levels of responding with the Tonend Light elements at test, as well as overshadowing of the Light byhe Tone observed on the first day of stage 2. Thus, RR appears toe obtained only when the competing stimulus is the more salientlement of the compound, as proposed by Liljeholm and Balleine12]. RR has actually been mostly described with extinction of the
ore salient element [8,9,13,33,34], but it may be noted that onlyhe study by Liljeholm and Balleine [12] and ours verified the lackf effect of manipulating the less salient element.
In Experiment 1, a control group allows us to separately assesshe effects of inflation or extinction. In this group, the Clicker stim-lus was never experienced in compound with the Light, so thatevaluation should be prevented. Comparison of the experimentalroups with this control group shows that RR occurred followingnflation, but not extinction, of the salient Tone, indicating the pres-nce of backward blocking, but not unovershadowing. This results original as backward blocking has rarely been observed in rats,nd so far only in aversive procedures [10,35–38]. Moreover, ourata contrast with those reported under a comparable procedure byiljeholm and Balleine [12] who observed only unovershadowing.he reasons for this discrepancy are presently unknown and war-ant further studies. However the results of our Experiment 1 wereeplicated during Experiment 2. In both studies, RR is only observedhen the more salient stimulus is manipulated in stage 2, suggest-
ng that the relative salience of the compound elements also plays role in backward blocking. This result is in agreement with theeneral ideas of Miller and co-workers, who suggested that a weakontrol of behaviour by the target cue (i.e. a low biological signifi-ance of the target) is a critical determinant of whether backwardlocking can be observed [10,35,38].
The primary objective of this study was not to decide betweenR theories. Still, our observation of backward blocking only underarticular conditions of stimulus salience leads to interesting theo-etical considerations. The existence of RR implies that some form
Research 223 (2011) 262– 270 267
of association develops between the elements of the compoundduring training [3–5]. To account for RR, Dickinson & Burke [3]proposed a modified version of Wagner’s [2] SOP model (mSOP).According to this theory, the presentation during stage 2 of oneelement (the competing stimulus, A) from the previous compoundstimulus (AX) evokes a representation (an A2 state) of the alternate,target stimulus (X), leading to changes in the latter’s associa-tive status. When both the target stimulus and the US are absent(extinction), they are both represented in the A2 state and theassociation between them is enhanced, leading to unovershad-owing. Conversely, if the competing stimulus undergoes inflation(see Fig. 6A), the associative strength of the target stimulus shoulddecrease because the actual US activates a different representa-tion (an A1 state). However, the inflation procedure could haveambiguous effects [39], because both the actual US (A1) and anevoked representation of the US (A2) may contribute in oppositeways to learning. Indeed, what is learned depends upon the differ-ence between A1 and A2 activations which represents a predictionerror. Moreover, a salient competing stimulus is likely to evoke astrong representation of the US in the A2 state, thereby counteract-ing the putative backward blocking effect. Hence, the mSOP doesnot clearly predict the backward blocking and therefore does notfit well with our data. Similar predictions are made by Van-Hammeand Wasserman’s revisited Rescorla–Wagner model [4]. Again, thismodel would expect backward blocking when the target is strongand the competing cue is weak.
An alternative account of RR is the comparator hypothesis [5,40].Unlike acquisition-based models, this model is based on the expres-sion of conditioned responses during retrieval. It assumes that thetarget presentation at test initiates a comparison process betweenthe strength of a direct activation of the US by the target stimulus Xand the strength of an indirect activation of the US representation,through associative links with the competing stimulus A (X-A andA-US). Specifically, extinction of A in stage 2 reduces the strength ofthe A-US association, leading to a decrease of the indirect activationof the US, thus enhancing responding to target X (unovershad-owing). Conversely, inflation of A reinforces the A-US association,leading to an increase of the indirect activation of the US relativeto the direct one, thereby decreasing the response to X (backwardblocking, see Fig. 6B). It is likely that this mechanism is all the morereliable when A is more salient than X, providing a potentiallystrong indirect activation of the US. The comparator hypothesistherefore predicts both unovershadowing and backward blocking.In particular, it accounts for our observation of backward blockingwith a salient competing stimulus.
The second part of our study aimed to identify the functionalneuronal circuit involved in RR, at present still poorly known.Experiment 2 compared c-Fos expression at the test phase in theInflation and Extinction groups, in response to the Light elementof the compound. Increased c-Fos immunoreactivity was found inthe infralimbic and orbitofrontal cortices of the group submittedto inflation. A similar difference, which did not reach significance,was also observed in the prelimbic cortex. Moreover, this Infla-tion group presented increased Fos expression within the nucleusaccumbens core and shell. It is noticeable that these increases in c-Fos expression were observed in the Inflation group, i.e. the groupthat expressed the fewer magazine entries at test, and are there-fore unlikely to reflect performance, such as conditioned responsesto the magazine. They suggest instead a cognitive control processthat actively inhibits responding at test. The lack of immunoreac-tivity difference in other brain areas such as sensory cortices andlimbic areas other than the nucleus accumbens suggests a selective
process involving a restricted neural circuit. Importantly, our pro-cedure equated exposure to all stimuli, including the food reward,so that groups Inflation and Extinction only differed by the rein-forcement history of the Tone stimulus during stage 2.
268 A. San-Galli et al. / Behavioural Brain Research 223 (2011) 262– 270
Fig. 6. Backward blocking according to the modified SOP and the comparator hypothesis. (A) According to the modified SOP [3], excitatory associations (grey arrows) developin stage 1 between representations of all the elements actually present (all in an A1 state), i.e. the target cue, X, the competing cue, A, and the US. During stage 2, presentationof the competing cue A evokes a representation of the target X in the A2 state via the A-X link, but also a representation of the US in the A2 state, via the A-US link. If thecompeting cue undergoes inflation, the evoked representation of the US (A2) and the actual US (A1) may induce opposite learning. Thus, the evoked-representation of targetX should form both an excitatory and an inhibitory connection (white arrow) with the US. A salient competing stimulus should evoke a strong representation of the US inthe A2 state, and counteract the putative backward blocking effect. (B) According to the comparator hypothesis [5], target presentation at test initiates a comparison processbetween a direct activation of the US by the target stimulus X and an indirect activation of the US representation, through associative links with the competing stimulus A( to an
X le wheU resen
rhdamtcpcopdiocaepmtiLtwttr
p
X-A and A-US). Inflation of A in stage 2 strengthens the A-US association, leading
therefore decreases (backward blocking). This mechanism should be more reliabS. Black circles: representations of stimuli physically present. Squares: evoked-rep
To our knowledge, our study is the first one to address the neu-al bases of RR in rats, as the only other available data come fromuman research. In a causal judgement task using fMRI, Corlett et al.escribed an activation of the right dorsolateral prefrontal cortexnd the ventral striatum [15,16]. Our results are in general agree-ent with this study as the rat medial prefrontal cortex, including
he prelimbic and infralimbic cortices, is considered functionallylose to the human dorsolateral prefrontal cortex [41]. A similarrefronto-accumbens circuit may therefore be involved in bothases. However, these activations were observed at different stagesf the procedures, and it is unsure whether they reflect the samerocess. In the study by Corlett et al. [15], the activations were seenuring stage 2 of the RR procedure, and were interpreted as reflect-
ng a learning process based on prediction error. As we focusedur analysis on the test phase, we cannot exclude that similar pro-esses occurred at stage 2 of our experiments. However, the c-Fosctivations that we observed may be related to the retrieval orxpression of previously acquired information, rather than to theresence of an error signal. Indeed, during stage 1, the Light ele-ent underwent overshadowing by the more salient Tone and was
herefore weakly conditioned. During stage 2, additional condition-ng to the Tone provided the animal with further evidence that theight element was not critical in the compound. As a consequence,he absence of food reward during test with the Light should in noay be surprising. According to this analysis, the inflation condi-
ion should involve little or no prediction error at test, suggesting
hat the neural activations reflect other processes underlying RR atetrieval.One such process could involve a comparison between inde-endently stored associations, as postulated by Miller and Matzel’s
increase of the indirect activation of the US relative to the direct one. Response ton A is more salient than X, providing a potentially strong indirect activation of thetations. Arrows: associative links.
comparator hypothesis. Our data suggest that this process reliesupon the activity of a prefronto–striatal circuit involving theinfralimbic and orbitofrontal cortices and the nucleus accumbens,and this assumption is supported by several lesion studies. Thebehavioural control exerted by inhibitory associations that com-pete with excitatory associations formed during another stage oflearning depends on the integrity of the infralimbic cortex, andits lesion was found to disrupt extinction [21,23], the retardationtest of a conditioned inhibition procedure [22] as well as reversallearning [42]. The involvement of the infralimbic cortex in back-ward blocking could therefore be understood by considering itsrole in mediating the expression of inhibitory control by compet-ing associations. During test, this region may thus be involved ininhibiting responding to the Light stimulus. Conversely, in groupExtinction, no c-Fos expression should be expected in the IL sinceextinction concerned the Tone stimulus in stage 2 and respondingto the Light at test was unaffected. The implication of the infral-imbic cortex in our task also fits with the alternate notion thatthis structure is concerned when different contingencies betweena stimulus and a reinforcer are established in separate learningphases. Indeed, following a two-stage procedure, performance ofrats with infralimbic lesions is dominated by the first-learnedcontingency, suggesting that the infralimbic cortex is needed toswitch between first-learned and second-learned contingencies[43]. Accordingly, one could predict that lesions of the infralim-bic cortex should affect RR by minimizing the impact of the second
phase of training. The orbitofrontal cortex is involved in encod-ing Pavlovian predictive relationship between cues and outcomesand generating outcome expectancies as suggested by lesion effectson Pavlovian-to-instrumental transfer, reinforcer devaluation and
Brain
dfmtsaItowiisScgviiccrg
uicmstioso
sbt[pc
A
gRlhHD
R
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
A. San-Galli et al. / Behavioural
ifferential outcome experiments [44,45]. It is therefore crucialor flexible behaviour such as reversal learning, in which subjects
ust change an established response in order to adapt to new con-ingency [18]. In RR, the orbitofrontal cortex might in particularupport the comparison process between independently learnedssociations necessary during retrieval of these associations at test.t would be interesting to determine whether the orbitofrontal cor-ex contributes to the retrieval of associations between cues andutcomes and of associations between elements of compound cueshich are necessary requirements for RR. The nucleus accumbens
s the recipient of information from a number of areas, including thenfralimbic and orbitofrontal cortices [46], and is considered a keyite mediating the ability of Pavlovian CSs to direct behaviour [47].tudies have shown that the integrity of the prefronto–accumbalircuit is required in tasks entailing shifts between rules and strate-ies, indicating that the nucleus accumbens is critical for gatingarious pathways contributing to behavioural flexibility [48–50]. Its therefore proposed that RR, and backward blocking in particular,s supported by retrieval and comparison of various Pavlovian asso-iations in the orbitofrontal cortex in combination with inhibitoryontrol occurring in the infralimbic cortex, and that these corticalegions exert a top-down influence over the nucleus accumbens toate Pavlovian responses [see 51].
Retrieval-based theories [5] suggest that similar processesnderlie both forward and backward blocking. In this context, it
s worth mentioning that recent findings [52] show that a cir-uit including the orbitofrontal cortex and the nucleus accumbensay also be required for forward unblocking processes. This study
pecifically demonstrated that the nucleus accumbens is requiredo overcome the effects of forward blocking through changesn either reward amount or identity. By contrast, the effects ofrbitofrontal cortex lesion were specific of identity unblocking,uggesting that this region may provide critical information aboututcome identity.
In conclusion, our data indicate that backward blocking can beeen in rats, in an appetitive magazine approach training. Backwardlocking could involve the integration of successive experiences athe retrieval stage of RR, as predicted in the comparator hypothesis5,40]. This process may be supported by neuronal activity within arefronto-striatal circuit involving the infralimbic and orbitofrontalortices as well as the nucleus accumbens shell and core.
cknowledgements
This work was supported by grants from the CNRS (Pro-ramme Interdisciplinaire Neuroinformatique) and Conseilégional d’Aquitaine. A. San-Galli was a fellow of the Ministère de
’Enseignement Supérieur. We are grateful to Dr. E. Coutureau foris valuable input to this study. We thank Pr. B. Balleine and Dr. N.olmes for their helpful comments on the manuscript, as well as. Panzeri, N. Argenta and J. Huard for animal care.
eferences
[1] Rescorla RA, Wagner AR. A theory of pavlovian conditioning: variations in theeffectiveness of reinforcement and nonreinforcement. In: Black AH, ProkasyWF, editors. Classical conditioning: current research and theory. New York:Appleton-Century-Crofts; 1972. p. 64–99.
[2] Wagner AR. SOP: A model of automatic memory processing in animal behavior.In: Spear NE, Miller RR, editors. Information processing in animals: memorymechanisms. Hillsdale, NJ: Erlbaum; 1981. p. 5–47.
[3] Dickinson A, Burke J. Within-compound associations mediate the retrospectiverevaluation of causality judgements. Q J Exp Psychol B 1996;49:60–80.
[4] Van Hamme LJ, Wasserman EA. Cue competition in causality judgments:
the role of nonrepresentation of compounds stimulus elements. Learn Motiv1994;25:127–51.[5] Miller RR, Matzel LD. The comparator hypothesis: A response rule for theexpression of associations. In: Bower GH, editor. The psychology of learningand motivation. San Diego: CA: Academic Press; 1988. p. 51–92.
[
[
Research 223 (2011) 262– 270 269
[6] Larkin MJ, Aitken MR, Dickinson A. Retrospective revaluation of causal judg-ments under positive and negative contingencies. J Exp Psychol Learn MemCogn 1998;24:1331–52.
[7] Wasserman EA, Castro L. Surprise and change: variations in the strength ofpresent and absent cues in causal learning. Learn Behav 2005;33:131–46.
[8] Kaufman MA, Bolles RC. A nonassociative aspect of overshadowing. Bull Psy-chon Soc 1981;18:318–20.
[9] Miller RR, Barnet RC, Grahame NJ. Responding to a conditioned stimulusdepends on the current associative status of other cues present during train-ing of that specific stimulus. J Exp Psychol Anim Behav Process 1992;18:251–64.
10] Miller RR, Matute H. Biological significance in forward and backward blocking:resolution of a discrepancy between animal conditioning and human causaljudgment. J Exp Psychol Gen 1996;125:370–86.
11] Shevill I, Hall G. Retrospective revaluation effects in the conditioned suppres-sion procedure. Q J Exp Psychol B 2004;57:331–47.
12] Liljeholm M, Balleine BW. Stimulus salience and retrospective revaluation. JExp Psychol Anim Behav Process 2006;32:481–7.
13] Sissons HT, Urcelay GP, Miller RR. Overshadowing and CS duration: counterac-tion and a reexamination of the role of within-compound associations in cuecompetition. Learn Behav 2009;37:254–68.
14] Holland PC. Overshadowing and blocking as acquistion deficits: no recov-ery after extinction of overshadowing or blocking cues. Q J Exp Psychol1999;52B:307–33.
15] Corlett PR, Aitken MR, Dickinson A, Shanks DR, Honey GD, Honey RA, et al.Prediction error during retrospective revaluation of causal associations inhumans: fMRI evidence in favor of an associative model of learning. Neuron2004;44:877–88.
16] Corlett PR, Murray GK, Honey GD, Aitken MR, Shanks DR, Robbins TW, et al. Dis-rupted prediction-error signal in psychosis: evidence for an associative accountof delusions. Brain 2007;130:2387–400.
17] de Wit S, Kosaki Y, Balleine BW, Dickinson A. Dorsomedial prefrontal cortexresolves response conflict in rats. J Neurosci 2006;26:5224–9.
18] Schoenbaum G, Roesch MR, Stalnaker TA, Takahashi YK. A new perspective onthe role of the orbitofrontal cortex in adaptive behaviour. Nat Rev Neurosci2009.
19] Naneix F, Marchand AR, Di Scala G, Pape JR, Coutureau E. A role for medialprefrontal dopaminergic innervation in instrumental conditioning. J Neurosci2009;29:6599–606.
20] Pickens CL, Saddoris MP, Gallagher M, Holland PC. Orbitofrontal lesionsimpair use of cue-outcome associations in a devaluation task. Behav Neurosci2005;119:317–22.
21] Rhodes SE, Killcross S. Lesions of rat infralimbic cortex enhance recoveryand reinstatement of an appetitive Pavlovian response. Learn Mem 2004;11:611–6.
22] Rhodes SE, Killcross AS. Lesions of rat infralimbic cortex result in disruptedretardation but normal summation test performance following training on aPavlovian conditioned inhibition procedure. Eur J Neurosci 2007;26:2654–60.
23] Rhodes SE, Killcross AS. Lesions of rat infralimbic cortex enhancerenewal of extinguished appetitive Pavlovian responding. Eur J Neurosci2007;25:2498–503.
24] Gluck MA, Meeter M, Myers CE. Computational models of the hippocampalregion: linking incremental learning and episodic memory. Trends Cogn Sci2003;7:269–76.
25] Coutureau E, Di Scala G. Entorhinal cortex and cognition. Prog Neuropsy-chopharmacol Biol Psychiatry 2009;33:753–61.
26] Balleine BW, Killcross S. Parallel incentive processing: an integrated view ofamygdala function. Trends Neurosci 2006;29:272–9.
27] Haluk DM, Floresco SB. Ventral striatal dopamine modulation of different formsof behavioral flexibility. Neuropsychopharmacology 2009;34:2041–52.
28] Paxinos G, Watson C. The rat brain in stereotaxic coordinates. Fourth EditionEdSan Diego, CA: Academic Press; 1998.
29] Balleine BW, Espinet A, Gonzalez F. Perceptual learning enhances retrospectiverevaluation of conditioned flavor preferences in rats. J Exp Psychol Anim BehavProcess 2005;31:341–50.
30] Speers MA, Gillan DJ, Rescorla RA. Within-compound associations in a varietyof compound conditioning procedures. Learn Motiv 1980;11:135–49.
31] Nakajima S, Kawai N. Failure of retrospective inference in rats’ taste aversion.Jpn Psychol Res 1997;39:87–97.
32] Pineno O. An examination of the effectiveness of inflation and deflation treat-ments in detecting within-compound learning of a taste aversion. BehavProcess 2007;75:33–9.
33] Matzel LD, Schachtman TR, Miller RR. Recovery of an overshadowed asso-ciation achieved by extinction of the overshadowing stimulus. Learn Motiv1985;16:398–412.
34] Denniston JC, Savastano HI, Blaisdell AP, Miller RR. Cue competition as aretrieval deficit. Learn Motiv 2003;34:1–31.
35] Denniston JC, Miller RR, Matute H. Biological significance as a determinant ofcue competition. Psychol Sci 1996;7:325–31.
36] Pineno O, Urushihara K, Miller RR. Spontaneous recovery from forward andbackward blocking. J Exp Psychol Anim Behav Process 2005;31:172–83.
37] Pineno O, Urushihara K, Stout S, Fuss J, Miller RR. When more is less: extendingtraining of the blocking association following compound training attenuatesthe blocking effect. Learn Behav 2006;34:21–36.
38] Urushihara K, Miller RR. Backward blocking in first-order conditioning. J ExpPsychol Anim Behav Process 2010;36:281–95.
2 Brain
[
[
[
[
[
[
[
[
[
[
[
[
[
70 A. San-Galli et al. / Behavioural
39] Aitken MR, Dickinson A. Simulations of a modified SOP model applied to retro-spective revaluation of human causal learning. Learn Behav 2005;33:147–59.
40] Stout SC, Miller RR. Sometimes-competing retrieval (SOCR): a formalization ofthe comparator hypothesis. Psychol Rev 2007;114:759–83.
41] Brown VJ, Bowman EM. Rodent models of prefrontal cortical function. TrendsNeurosci 2002;25:340–3.
42] Chudasama Y, Robbins TW. Dissociable contributions of the orbitofrontal andinfralimbic cortex to pavlovian autoshaping and discrimination reversal learn-ing: further evidence for the functional heterogeneity of the rodent frontalcortex. J Neurosci 2003;23:8771–80.
43] George DN, Duffaud AM, Pothuizen HH, Haddon JE, Killcross AS. Lesions to theventral, but not the dorsal, medial prefrontal cortex enhance latent inhibition.Eur J Neurosci 2010;31:1474–82.
44] Ostlund SB, Balleine BW. The contribution of orbitofrontal cortex to actionselection. Ann N Y Acad Sci 2007;1121:174–92.
45] Delamater AR. The role of the orbitofrontal cortex in sensory-specific encodingof associations in pavlovian and instrumental conditioning. Ann N Y Acad Sci2007;1121:152–73.
[
Research 223 (2011) 262– 270
46] Zahm DS, Brog JS. On the significance of subterritories in the “accumbens” partof the rat ventral striatum. Neuroscience 1992;50:751–67.
47] Cardinal RN, Parkinson JA, Hall J, Everitt BJ. Emotion and motivation: the role ofthe amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav Rev2002;26:321–52.
48] Goto Y, Grace AA. Dopaminergic modulation of limbic and cortical drive ofnucleus accumbens in goal-directed behavior. Nat Neurosci 2005;8:805–12.
49] Goto Y, Grace AA. Limbic and cortical information processing in the nucleusaccumbens. Trends Neurosci 2008;31:552–8.
50] Block AE, Dhanji H, Thompson-Tardif SF, Floresco SB. Thalamic-prefrontalcortical-ventral striatal circuitry mediates dissociable components of strategyset shifting. Cereb Cortex 2007;17:1625–36.
51] Jung YC, Ku J, Namkoong K, Lee W, Kim SI, Kim JJ. Human orbitofrontal-striatum
functional connectivity modulates behavioral persistence. Neuroreport2010;21:502–6.52] McDannald MA, Lucantonio F, Burke KA, Schoenbaum GS. Ventral striatum andorbitofrontal cortex are both required for model-based, but not model-free,reinforcement learning. J Neurosci 2011;31:2700–5.