hold your horses: a dynamic computational role for the...

Frank 1

Hold Your Horses:A Dynamic Computational Role for the Subthalamic Nucleus in Decision Making

Michael J. FrankDept of Psychology and Program in Neuroscience

University of Arizona1503 E University Blvd

Tucson, AZ [email protected]

Abstract:

The basal ganglia (BG) coordinate decision making processes by facilitating adaptive frontal motor commands whilesuppressing others. In previous work, neural network simulations accounted for response selection deficits associatedwith BG dopamine depletion in Parkinson’s disease. Novel predictions from this model have been subsequentlyconfirmed in Parkinson patients and in healthy participants under pharmacological challenge. Nevertheless, one clearlimitation of that model is in its omission of the subthalamic nucleus (STN), a key BG structure that participates inboth motor and cognitive processes. The present model incorporates the STN and shows that by modulating when aresponse is executed, the STN reduces premature responding and therefore has substantial effects on which responseis ultimately selected, particularly when there are multiple competing responses. Increased cortical response conflictleads to dynamic adjustments in response thresholds via cortico-subthalamic-pallidal pathways. The model accuratelycaptures the dynamics of activity in various BG areas during response selection. Simulated dopamine depletion resultsin emergent oscillatory activity in BG structures, which has been linked with Parkinson’s tremor. Finally, the modelaccounts for the beneficial effects of STN lesions on these oscillations, but suggests that this benefit may come at theexpense of impaired decision making.

Introduction

Deciphering the mechanisms by which the brain sup-ports response selection, a central process in decisionmaking, is an important challenge for both the artifi-cial intelligence and cognitive neuroscience communities.Based on a wealth of data, the basal ganglia (BG) arethought to play a principle role in these processes. In thecontext of motor control, various authors have suggestedthat the role of the BG is to selectively facilitate the ex-ecution of a single adaptive motor command, while sup-pressing all others (Mink, 1996; Hikosaka, 1994; Basso &Wurtz, 2002; Jiang, Stein, & McHaffie, 2003; Brown,Bullock, & Grossberg, 2004; Redgrave, Prescott, & Gur-ney, 1999; Gurney, Prescott, & Redgrave, 2001; Frank,2005a). Interestingly, circuits linking the BG with morecognitive areas of frontal cortex (e.g., prefrontal) arestrikingly similar to those observed in the motor domain(Alexander, DeLong, & Strick, 1986), raising the possi-

Neural Networks, in press. (Special Issue on the Neurobiol-ogy of Decision Making). Portions of this paper were previouslypresented in conference format at the International Workshopon Models of Natural Action Selection (Frank, 2005b). I thankRandy O’Reilly, Adam Aron, Patrick Simen, Todd Braver andJonathan Cohen for helpful discussion.

bility that the BG participate in cognitive decision mak-ing in an analogous fashion to their role in motor control(Beiser & Houk, 1998; Middleton & Strick, 2000, 2002;Frank, Loughry, & O’Reilly, 2001; Frank, 2005a; Frank &Claus, in press). studies with Parkinson’s patients, whohave severely depleted levels of dopamine (DA) in theBG (Kish, Shannak, & Hornykiewicz, 1988), have pro-vided insights into the functional roles of the BG/DA sys-tem in both motor and higher level cognitive processes(Cools, 2005; Frank, 2005a; Shohamy, Myers, Grossman,Sage, & Gluck, 2005a). Of particular recent interest isthe finding that deep brain stimulation in the subthala-mic nucleus (STN) dramatically improves Parkinson mo-tor symptoms, with both reported enhancements and im-pairments in cognition (Karachi, Yelnik, Tande, Trem-blay, Hirsch, & Francois, 2004; Witt, Pulkowski, Herzog,Lorenz, Hamel, Deuschl, & Krack, 2004). Because theBG consists of a complex network of dynamically inter-acting brain areas, a mechanistic understanding of exactlyhow the STN participates in response selection and deci-sion making is difficult to develop with traditional box andarrow models. Computational models that explore the dy-namics of BG network activity are therefore useful toolsfor providing insight into these issues, and in turn, howthey affect individuals with PD and related disorders.

2 A Dynamic Computational Role for the Subthalamic Nucleus in Decision Making

In this paper, I review converging evidence for amechanistic, functional account of how interacting ar-eas within the BG-frontal system learn to select adaptiveresponses and participate in cognitive decision making,as informed by prior computational simulations. I thenpresent a neural network model that explores the uniquecontribution of the STN within the overall BG circuitry.The simulations reveal that the STN can dynamically con-trol the threshold for executing a response, and that thisfunction is adaptively modulated by the degree to whichmultiple competing responses are activated, as in difficultdecisions. It is concluded that the STN may be essentialto allow all information to be integrated before makingdecisions, and thereby prevents impulsive or prematureresponding during high-conflict decision trials. Further-more, analysis of the dynamics of activity within variousBG areas during response selection in intact and simu-lated Parkinson states demonstrates a striking relationshipto the same patterns observed physiologically, providingsupport for the model’s biological plausibility and furtherinsight into the neural processes underlying response se-lection.

Overall BG Network Functionality

The “standard model” proposes that two BG pathwaysindependently act to selectively facilitate the execution ofthe most appropriate cortical motor command, while sup-pressing competing commands (Albin, Young, & Penney,1989; Mink, 1996). Two main projection pathways fromthe striatum go through different BG nuclei on the wayto thalamus and up to cortex (Figure 1a). Activity in thedirect pathway sends a “Go” signal to facilitate the execu-tion of a response considered in cortex, whereas activityin the indirect pathway sends a “NoGo” signal to suppresscompeting responses (Alexander & Crutcher, 1990a; Ger-fen & Wilson, 1996). More specifically, striatal Go cellsdirectly project to and inhibit the internal segment of theglobus pallidus (GPi), which in turn disinhibits the thala-mus, ultimately facilitating the execution of cortical mo-tor responses. Conversely, striatal NoGo cells project toand inhibit the external segment of the globus pallidus(GPe), releasing the tonic inhibition of GPe onto GPi,and therefore having an opposing effect on motor activity.Dopamine modulates the relative balance of these path-ways by exciting synaptically-driven activity in Go cellsvia D1 receptors, while inhibiting NoGo activity via D2receptors (Gerfen, 1992; Aubert et al., 2000; Hernandez-Lopez et al., 1997; Hernandez-Lopez et al., 2000; Joel &Weiner, 1999; Brown et al., 2004; Frank, 2005a). Physio-logical evidence for this model comes from studies show-ing opposite effects of D1 and D2 agents on activitywithin the two types of cells and BG output nuclei (Ger-

fen, 2000; Salin, Hajji, & Kerkerian-Le Goff, 1996; Bo-raud, Bezard, Bioulac, & Gross, 2002; Robertson, Vin-cent, & Fibiger, 1992; Gerfen, Keefe, & Gauda, 1995).Moreover, the general aspects of this model have beensuccessfully leveraged to explain various motor deficitsobserved in patients with BG dysfunction (e.g., Albinet al., 1989).

Recently, several researchers have pointed out that thesimplest version of the standard BG models is inadequate,and that a more advanced dynamic conceptualization ofBG function is required, motivating the use of computa-tional modeling (Gurney et al., 2001; Bar-Gad, Morris, &Bergman, 2003; Brown et al., 2004; Frank, 2005a). Oth-ers question the most basic assumptions of the standardmodel, suggesting that the segregation of the “direct” and“indirect” BG pathways is not as clear as once thought(Kawaguchi, Wilson, & Emson, 1990; Wu, Richard, &Parent, 2000; Levesque & Parent, 2005). These studiesfound that rather than the striatum having separate pop-ulations projecting to GPi and GPe, virtually all striatalcells projected to GPe, while there was still a subpopu-lation that also projected to GPi. While these data cer-tainly challenge the simplest version of the direct/indirectmodel, one need not “throw the baby out with the bathwa-ter” and reject the standard model altogether, especiallygiven its contribution to many theoretical and practical ad-vances. First, putative NoGo cells remain evident in theabove studies, in that many cells only projected to GPeand not GPi. Second, the possibility that Go cells project-ing to GPi also project to GPe may signify that rather thancomputing raw Go signals, the BG output may computethe temporal derivative of these signals. According to thisscheme, a striatal Go signal would first inhibit the GPiand disinhibit the thalamus, thereby facilitating the corti-cal response as proposed by the standard model. Concur-rently, the same striatal activity would inhibit GPe, oppos-ing the initial facilitation via GPe-GPi disinhibition. Im-portantly, this opposing signal would be temporally de-layed relative to the initial facilitation (given the extrasynapse and slower time constant associated with disin-hibition). The net result would be that Go signals lead toshort-lasting GPi pauses and associated thalamic burstingactivity. This overall functionality may be useful for rapidsequencing of several sub-motor commands, in which anindividual command should be facilitated and then imme-diately suppressed in favor of the subsequent command.Consistent with this idea, motor cortical neurons that sendtheir main axons to the pyramidal tract (and therefore di-rectly involved in movement) also send collaterals only toNoGo-type striatal cells (Lei, Jiao, Del Mar, & Reiner,2004), which would then act to suppress/terminate themovement. Overall, while some of the anatomical detailsare undoubtedly missing from any computational modelof the BG, this simplification enables analysis of the dy-

Frank 3

namics of activity among multiple BG regions which isnot feasible with detailed but static anatomical diagrams.

A Model of Reinforcement Learning and Deci-sion Making in Parkinson’s Disease

Studies with patients with Parkinson’s disease (PD)provide insight into the response selection / decisionmaking functions of the basal ganglia. PD is char-acterized by death of midbrain dopamine cells project-ing to the BG (Kish et al., 1988), and associated mo-tor symptoms including tremor, rigidity, and slownessof movement (McAuley, 2003). PD patients also havea variety of cognitive deficits, ranging from procedu-ral learning to working memory, decision making andattention (Brown & Marsden, 1988; Knowlton, Man-gels, & Squire, 1996; Jackson, Jackson, Harrison, Hender-son, & Kennard, 1995; Ashby, Noble, Ell, Filoteo, & Wal-dron, 2003; Maddox & Filoteo, 2001; Gotham, Brown, &Marsden, 1988; Rogers, Sahakian, Hodges, Polkey, Ken-nard, & Robbins, 1998; Cools, Barker, Sahakian, &Robbins, 2001b, 2003a; Shohamy, Myers, Grossman,Sage, & Gluck, 2005b; Frank, 2005a; Frank, Seeberger, &O’Reilly, 2004). Given the proposed BG role in selectingamong various competing low-level motor responses bymodulating frontal motor activity, and the parallel circuitslinking the BG with more frontal cognitive areas (Alexan-der et al., 1986; Middleton & Strick, 2000, 2002), it is nat-ural to extend this action selection functionality to includehigher-level cognitive decisions (Frank et al., 2001; Frank,2005a; Houk, 2005; Frank & Claus, in press). While oth-ers explore BG mechanisms that lead to adaptive selectionof a salient response in the face of less salient competi-tors (e.g., Gurney et al., 2001; Gurney, Humphries, Wood,Prescott, & Redgrave, 2004), a complementary questionis how the BG learn which action to select? This questionis relevant because the most salient input may not alwaysbe the best choice.

Previous computational modeling of the basal ganglia/ dopamine system provided an explicit formulation thatattempts to address this question and ties together vari-ous cognitive deficits in Parkinson’s disease (PD) (Frank,2005a). Specifically, the model (Figure 1b) posits thatphasic changes in DA during error feedback are criti-cal for modulating Go/NoGo representations in the BGthat facilitate or suppress the execution of motor com-mands. The main assumption was that during positiveand negative feedback (e.g., when participants are toldthat their responses were correct or incorrect), bursts anddips of DA occur that drive learning about the responsejust executed. This assumption was motivated by a largeamount of evidence for bursts and dips of DA during re-wards or their absence in animals, respectively (Schultz,Dayan, & Montague, 1997; Schultz, 2002; Satoh, Nakai,

Sato, & Kimura, 2003; Bayer & Glimcher, 2005; Pan,Schmidt, Wickens, & Hyland, 2005). These DA changeshave also been inferred to occur in humans receivingpositive and negative feedback in cognitive tasks (Hol-royd & Coles, 2002; Delgado, Locke, Stenger, & Fiez,2003; Frank, Woroch, & Curran, 2005). Moreover, thesephasic changes in DA modulate neuronal excitability, andmay therefore act to reinforce the efficacy of recently ac-tive synapses (e.g., Hebb, 1949; see Mahon et al. (2003)for evidence of Hebbian learning in striatum), leading tothe learning of rewarding behaviors. In the model, “cor-rect” responses are followed by transient increases in sim-ulated DA that enhance synaptically-driven activity in thedirect/Go pathway via simulated D1 receptors, while con-currently suppressing the indirect/NoGo pathway via sim-ulated D2 receptors (for detailed neurobiological support,see Frank, 2005a; Brown et al., 2004; Frank & O’Reilly,in press). This drives Go learning, enabling the model toselect responses that on average result in positive feed-back. Conversely, phasic dips in DA following incorrectresponses release NoGo neurons from the suppressive in-fluence of DA, allowing them to be further excited by cor-ticostriatal glutamate, and driving NoGo learning.1 With-out ever having access to a supervised training signal as towhich response should have been selected, over the courseof training intact networks nevertheless learned how torespond in probabilistic classification tasks, similarly tohealthy participants. When 75% of units in the SNc DAlayer of the model were lesioned to simulate the approx-imate amount of damage in PD patients, the model wasimpaired similarly to patients.

The details of the BG model are described in Frank(2005). In brief, the premotor cortex represents and “con-siders” two possible responses (R1 and R2) for each inputstimulus. The BG system modulates which one of theseresponses is facilitated and which is suppressed by sig-naling Go or NoGo to each of the responses. The fourcolumns of units in the striatum represent, from left toright, Go-R1, Go-R2, NoGo-R1 and NoGo-R2. In the ab-sence of synaptic input, GPi and GPe units are tonicallyactive. Go and NoGo representations for each responsecompete at the level of GPi, such that stronger Go repre-sentations lead to disinhibition of the corresponding col-umn of the thalamus, which in turn amplifies and facil-itates the execution of that response in premotor cortex.Concurrently, the alternative response is suppressed.

Striatal Go/NoGo representations are learned via pha-sic changes in simulated DA firing in the SNc layer dur-ing positive and negative reinforcement. After correct re-sponses, increases in DA firing excite Go units for the

1See Frank and O’Reilly (in press) for more biological justification,including discussion on how DA dips can be effective learning signalsdespite the already low tonic firing rates of DA neurons.


a) b)Frontal Cortex

GPe

STND2−

D1+

excitatoryinhibitory

Go NoGo thalamus

GPi

excitatoryinhibitorymodulatory

SNc

striatum

Input

Striatum

SNc

Thalamus

Output

−

−−Go NoGo

−

R1 R2

Premotor Cortex

D1 D2

GPi

GPe

Figure 1: a) The striato-cortical loops, including the direct(“Go”) and indirect (“NoGo”) pathways of the basal ganglia.The Go cells disinhibit the thalamus via GPi, thereby facilitat-ing the execution of an action represented in cortex. The NoGocells have an opposing effect by increasing inhibition of the tha-lamus, suppressing actions from getting executed. Dopaminefrom the SNc projects to the dorsal striatum, causing excitationof Go cells via D1 receptors, and inhibition of NoGo via D2receptors. GPi: internal segment of globus pallidus; GPe: ex-ternal segment of globus pallidus; SNc: substantia nigra parscompacta; STN: subthalamic nucleus. b) The Frank (2005) neu-ral network model of this circuit (squares represent units, withheight reflecting neural activity). The Premotor Cortex selectsan Output response via direct projections from the sensory In-put, and is modulated by the BG projections from Thalamus.Go units are in the left half of the Striatum layer; NoGo in theright half, with separate columns for the two responses (R1 andR2). In the case shown, striatum Go is stronger than NoGo forR1, inhibiting GPi, disinhibiting Thalamus, and facilitating R1execution in cortex. A tonic level of dopamine is shown in SNc;a burst or dip ensues in a subsequent error feedback phase (notshown), driving Go/NoGo learning. The contributions of theSTN were omitted from this model, but are explored in the cur-rent simulations (Figure 2).

just-selected response, while suppressing NoGo units, viasimulated D1 and D2 receptors. Conversely, decreases inDA after incorrect responses, together with corticostriatalglutamate release, results in increased NoGo activity forthat response. This DA modulation of Go/NoGo activitydrives learning as described above.

As DA bursts and dips reinforce Go and NoGo rep-resentations in the BG, our model showed that the mostadaptive (i.e., rewarding) responses represented in premo-tor areas will tend to get facilitated while less adaptiveones are suppressed. Further, as the BG learns to facil-itate adaptive responses, the associated premotor repre-sentations become enhanced (via Hebbian learning). Inthis way, DA reward processes within the BG may ingrainprepotent motor “habits” in frontal cortical areas (Frank,2005a; Frank & Claus, in press). Once these habits are in-grained, there is less need for selective facilitation by theBG. This is consistent with observations that dopaminer-gic integrity within the BG is much more critical for the

acquisition rather than the execution of instrumental re-sponses (Smith-Roe & Kelley, 2000; Parkinson, Dalley,Cardinal, Bamford, Fehnert, Lachenal, Rudarakanchana,Halkerston, Robbins, & Everitt, 2002; Choi, Balsam, &Horvitz, 2005), and with recent physiological observa-tions that learning-related activity is initially seen in theBG, and is only observed later in frontal cortex (Delgado,Miller, Inati, & Phelps, 2005; Pasupathy & Miller, 2005).

Modeling Dopaminergic Medication Effects onCognitive Function in PD

The same model was used to explain certain nega-tive effects of dopaminergic medication on cognition inPD (Frank, 2005a). While medication improves cog-nitive performance in some attentional tasks (Swainson,Rogers, Sahakian, Summers, Polkey, & Robbins, 2000;Cools, Barker, Sahakian, & Robbins, 2001a; Shohamyet al., 2005b), it actually impairs performance in prob-abilistic reversal learning (Swainson et al., 2000; Coolset al., 2001a; Cools, 2005), that is when having to makedecisions that require learning to overriding previouslyadaptive responses in favor of those that were less adap-tive.

In order to simulate medication effects, it was hypoth-esized that medication increases the tonic level of DA,but that this interferes with the natural biological sys-tem’s ability to dynamically regulate phasic DA changes.Specifically, phasic DA dips during negative feedbackmay be partially blocked by DA agonists (or increasesin tonic DA by L-Dopa; Pothos, Davila, & Sulzer, 1998)that continue to bind to receptors. When this was simu-lated in the model, selective deficits were observed dur-ing probabilistic reversal, despite equivalent performancein the acquisition phase (Frank, 2005a), mirroring theresults found in medicated patients. Because increasedtonic levels of DA suppressed the indirect/NoGo path-way, networks were unable to learn “NoGo” to overridethe prepotent response learned in the acquisition stage.This account is consistent with similar reversal deficitsobserved in healthy participants administered an acutedose of bromocriptine, a D2 agonist (Mehta, Swainson,Ogilvie, Sahakian, & Robbins, 2001), and with severalother learning deficits induced by DA medications that areconsistent with NoGo impairments (Ridley, Haystead, &Baker, 1981; Smith, Neill, & Costall, 1999; Charbon-neau, Riopelle, & Beninger, 1996; Czernecki, Pillon,Houeto, Pochon, Levy, & Dubois, 2002; Cools, Barker,Sahakian, & Robbins, 2003b; Shohamy, Myers, Gegh-man, Sage, & Gluck, in press; Bokura, Yamaguchi, &Kobayashi, 2005; Frank et al., 2004).

Frank 5

Empirical Tests of the ModelRecently, we have tested various aspects of the hy-

pothesized roles of the basal ganglia / dopamine sys-tem across both multiple cognitive processes. First, wedemonstrated support for a central prediction of the Frank(2005a) model regarding BG dopamine involvement in“Go” and “NoGo” cognitive reinforcement learning. Wetested Parkinson patients on and off medication, alongwith healthy senior control participants (Frank et al.,2004). We predicted that decreased levels of dopamine inParkinson’s disease would lead to spared NoGo learning,but impaired Go learning (which depends on DA bursts).We further predicted that dopaminergic medication shouldalleviate the Go learning deficit, but would block the ef-fects of dopamine dips needed to support NoGo learning.Results were consistent with these predictions. In a prob-abilistic learning task, all patients and aged-matched con-trols learned to make choices that were more likely to re-sult in positive rather than negative reinforcement. Thedifference was in their learning biases: patients takingtheir regular dose of dopaminergic medication implicitlylearned more about the positive outcomes of their deci-sions (i.e., they were better at Go learning), whereas thosewho had abstained from taking medication implicitlylearned to avoid negative outcomes (better NoGo learn-ing). Age-matched controls did not differ in their ten-dency to learn more from the positive/negative outcomesof their decisions. We have also found the same patternin young healthy participants administered dopamine D2receptors agonists and antagonists, which at low dosesmodulate striatal dopamine release (Frank & O’Reilly, inpress). Again, dopamine increases improved Go learn-ing and impaired NoGo learning, while decreases had theopposite effect. The same BG modeling framework ac-curately predicted the pattern of event-related potentialsrecorded from healthy participants who were biased tolearn more from either positive or negative reinforcement(Frank et al., 2005), as well as a counter-intuitive im-provement in BG/DA-dependent choices when hippocam-pal explicit memory systems were taken offline by thedrug midazolam (Frank, OReilly, & Curran, in press). Fi-nally, in the D2 drug study mentioned above, the sameBG/DA effects extended to higher level working memorytasks that required paying attention to task-relevant (i.e.,positively valenced) information while ignoring distract-ing (negative) information (Frank & O’Reilly, in press),consistent with predictions from extended BG modelsthat include interactions with prefrontal cortex in work-ing memory and attention (Frank et al., 2001; O’Reilly &Frank, 2006; Frank & Claus, in press).

Input

Striatum

GP Int

GP Ext

Thalamus

STN

D1

GO NOGO

D2

R2R1 R3 R4

R2R1 R3 R4R2R1 R3 R4

Output (M1)

SNc (DA)

Premotor Cortex

Figure 2: The subthalamic nucleus is incorporated into ascaled-up model that includes four competing responses (R1-R4). The STN receives excitatory projections from pre/motorcortex in the “hyperdirect pathway” and excites both GPi andGPe; GPe provides inhibitory feedback on STN activity.

Integrating Contributions of the SubthalamicNucleus in the Model

Despite its success in capturing dopamine-driven in-dividual differences in learning and attentional processes,the above model falls short in its ability to provide insightinto BG dynamics that depend on the subthalamic nucleus(STN). The model was designed to simulate how the BGcan learn to selectively facilitate (Go) one response whileselectively suppressing (NoGo) another. Because the pro-jections from the STN to BG nuclei (GPe and GPi) arediffuse (Mink, 1996; Parent & Hazrati, 1995), it may notbe well suited to provide selective (focused) modulationof specific responses, and was therefore omitted from themodel. Instead the model simulated the focused projec-tions from striatum to GPi and GPe, as well as the focusedprojections from GPe to GPi, to demonstrate how directand indirect pathways may compete with one another atthe level of each response, but may act in parallel to facil-itate and suppress alternative responses (see Frank, 2005for details and discussion).

Nevertheless, there is substantial evidence that theSTN is critically involved in both motor control and cog-nitive processes (Bergman, Wichmann, Karmon, & De-Long, 1994; Boraud et al., 2002; Baunez, Humby, Eagle,Ryan, Dunnett, & Robbins, 2001; Karachi et al., 2004;Witt et al., 2004). Further, other computational models ofaction selection also implicate a key role of the STN (Gur-ney et al., 2001; Rubchinsky, Kopell, & Sigvardt, 2003;


Brown et al., 2004). The present model explored the con-tributions of the STN within the computational frameworkof the previous model of cognitive reinforcement learningand decision making (Frank, 2005a), scaled up to includefour competing responses. By virtue of its diffuse con-nectivity to BG nuclei, the STN may support more of aglobal modulatory signal on facilitation and suppressionof all responses, rather than modulating the execution ofany particular response. The simulations described be-low reveal that this global modulatory signal could not bereplaced by a simple response threshold parameter, be-cause its effects are dynamic as response selection pro-cesses evolve, and its efficacy depends on excitatory inputfrom premotor cortex. Further, simulated dopamine de-pletion in the augmented model results in emergent os-cillations in the STN and BG output structures, whichhave been documented empirically and are thought to bethe source of Parkinson’s tremor. Finally, the simulationsshow that the STN may be critical for action selection pro-cesses to prevent premature responding, so that all poten-tial responses are considered before facilitating the mostappropriate one.

STN Connectivity with Other BG and CorticalStructures

The STN was included in the model in accordancewith known constraints on its connectivity in BG circuitry,as depicted in Figure 2. First, the STN forms part of the“hyperdirect” pathway, so-named because cortical activ-ity targets the STN, which directly excites GPi, bypassingthe striatum altogether (Nambu, Tokuno, Hamada, Kita,Imanishi, Akazawa, Ikeuchi, & Hasegawa, 2000). Thusinitial activation of the STN by cortex leads to an initialexcitatory drive on the already tonically active GPi, effec-tively making the latter structure more inhibitory on thethalamus, and therefore less likely to facilitate a response.Further, the STN gets increasingly excited with increasingcortical activity. Thus, if several competing responses areactivated, the STN sends a stronger “Global NoGo” signalwhich allows the BG system to fully consider all possibleoptions before sending a Go signal to facilitate the mostadaptive one.

Second, the STN and GPe are reciprocally connectedin a negative feedback loop, with the STN exciting theGPe and the GPe inhibiting the STN (Parent & Hazrati,1995). As noted above, the connections from STN to GPeare diffuse, and therefore are not likely to be involved insuppressing a specific response. Of the STN neurons thatproject to GPe, the vast majority also project to GPi (Sato,Parent, Levesque, & Parent, 2000). In the model, eachSTN neuron receives projections from two randomly se-lected GPe neurons. This was motivated by data show-ing that multiple GPe neurons converge on a single STN

neuron (Karachi et al., 2004). In contrast, each GPe neu-ron receives diffuse projections from all STN neurons (butwith randomly different synaptic weights). Please see Ap-pendix for additional model equations and parameters.

Simulated BG Firing Patterns During ResponseSelection

The firing patterns of simulated BG, thalamic, andcortical structures are shown in a representative responseselection trial in Figure 3b. Upon presentation of a stim-ulus input, multiple competing responses are simultane-ously but weakly activated in premotor cortex. Concur-rently, response-specific striatal NoGo signals cause GPeactivity to decrease. The combined effects of initial corti-cal activity and decreases in GPe activity produce an ini-tial STN surge at approximately 20 cycles of network set-tling (in this particular trial). This STN activity is exci-tatory onto all GPi cells, preventing them from gettinginhibited by early striatal Go signals that would other-wise facilitate response execution. However, STN activityalso excites GPe neurons, which in turn reciprocally in-hibit the initial STN activity surge, thereby removing theGlobal NoGo signal. At this point, a striatal Go signal fora particular response can then inhibit the correspondingGPi column, resulting in thalamic disinhibition and subse-quent selection of that response in motor cortex. Becauseactivity values are displayed in terms of average activityacross each layer, the selection of a single motor responsetogether with suppression of other responses results in anet decrease in average premotor cortex activity. Finally,in some trials, a late striatal NoGo signal causes GPe in-hibition and a second surge in STN activity.

The above description of STN dynamics is consis-tent with data from physiological recordings showing anearly discharge in STN cells during either response se-lection or direct cortical stimulation (Wichmann et al.,1994; Kolomiets et al., 2001; Nambu et al., 2000; Mag-ill et al., 2004; Figure 3a). Moreover, this model is aformal implementation of existing theoretical constructsregarding the role of the STN in initial response suppres-sion, followed by a direct pathway response facilitation,and then finally an indirect pathway response termination(Maurice, Deniau, Glowinski, & Thierry, 1998a; Nambu,Kaneda, Tokuno, & Takada, 2002). This dynamic func-tionality of BG activity in response selection may haveimportant implications for higher level decision making,as described below. But first, an obvious question iswhether this model also accounts for patterns of activity inthe dopamine-depleted state, for which there is abundantdata.

Frank 7

a) b)

Magill et al, 2004

STN act

0 20 40 60 80 100Cycle

0.00.10.20.30.40.50.60.70.80.91.0

Avg

Activ

ity L

evel

STNGPeGPiMotor CtxThalamus

Avg Activity During Selection:Single Trial

c) d)

Levy et al, 2000STNact

DA Depletion & Oscillations:

0 20 40 60 80 100Cycle

0.00.10.20.30.40.50.60.70.80.91.0

Avg

Activ

ity L

evel


Single Trial

Figure 3: a) Physiological STN recordings following cortical stimulation, adapted from Magill et al (2004), showing two bursts ofactivity. The same patterns are observed during natural response selection (Wichmann et al, 1994). b) A single response selectiontrial in the model. Activity levels (normalized with respect to maximal firing rates) are averaged across units within each layer as afunction of network settling cycles. STN activity is shaded in grey for comparison with a). Initially, multiple simultaneously activeand competing premotor cortex responses excite STN via the hyperdirect pathway (≈ cycle 20). The resulting “Global NoGo”signal prevents premature responding by keeping GPi units tonically active. Sustained GPe activity subsequently inhibits STN(cycle 35), turning off the Global NoGo signal. Striatal Go signals then facilitate a response by inhibiting GPi and disinhibitingThalamus (cycle 40). This is reflected in premotor cortex as an overall decrease in activity, due to suppression of the three alternativeresponses. Finally, striatal NoGo signals inhibit GPe, causing a second STN surge (cycle 60), thought to terminate the executedresponse. c) Dopamine depletion in animals and humans leads to oscillatory activity in STN (shown here from Levy et al, 2000)and GPe and GPi (not shown), which are associated with Parkinson’s tremor. d) Simulated dopamine depletion in the model leadsto emergent network oscillations in STN, GPi and GPe.


a) b)

0 10 20 30 40 50 60 70 80 90 100Cycle

0.00.10.20.30.40.50.60.70.80.91.0

Avg

Activ

ity L

evel

DA Depletion & Oscillations:Multiple Trials

0 10 20 30 40 50 60 70 80 90 100Cycle

0.00.10.20.30.40.50.60.70.80.91.0

Avg

Activ

ity L

evel


DA Depletion + STN Lesion:Multiple Trials

Figure 4: Average unit activity during response selection as afunction of network settling cycles. Data are averaged acrossunits within each area, and across 100 trials. a) SimulatedParkinsonism (DA depletion) led to oscillations in STN, GPeand GPi, as is observed in DA-depleted animals. Regular os-cillations are observed despite averaging across multiple trials,but are dampened relative to those observed in any single trial(e.g., Figure 3d). b) STN lesions in DA-depleted networks elim-inated the oscillations observed in GPe and GPi, and facilitatedresponse selection, as has been observed in experimental ani-mals (Bergman et al, 1990; Ni et al, 2000).

Dopamine Depletion is Associated with Subtha-lamic and Pallidal Oscillations

Dopaminergic depletion in Parkinson’s disease is as-sociated with changes in the firing patterns and activ-ity levels in various BG nuclei (Mink, 1996; Boraudet al., 2002). Lowered dopamine levels result in exces-sive striatal NoGo (indirect pathway) activity, and con-comitant decreases in GPe and increases in GPi activity(Boraud et al., 2002). Parkinsonism is also associatedwith increased STN activity, thought to arise from reducedGABAergic GPe input (Miller & Delong, 1987; DeLong,1990). Perhaps most notably, DA depletion has been reli-ably associated with low-rate oscillatory bursting activityin STN, GPe and GPi, which is correlated with the de-velopment of Parkinson’s tremor (Bergman et al., 1994;Bergman, Feingold, Nini, Raz, Slovin, Abeles, & Vaa-dia, 1998; Levy, Hutchison, Lozano, & Dostrovsky, 2000;Raz, Vaadia, & Bergman, 2000). Finally, these oscilla-tions and associated PD symptoms are eliminated in DA-depleted animals after they are given experimental STNlesions (Ni, Bouali-Benazzouz, Gao, Benabid, & Benaz-zouz, 2000; Bergman, Wichmann, & DeLong, 1990).2

Interestingly, when Parkinson’s disease was simu-lated in the model, these effects emerged naturally (Fig-

2The therapeutic effects of human STN deep brain stimulation arethought to rely on similar mechanisms (Benazzouz & Hallett, 2000;Meissner, Leblois, Hansel, Bioulac, Gross, Benazzouz, & Boraud,2005).

ures 3d) and 4a for averaged activity across multiple tri-als). First, simulated DA depletion (setting tonic SNc fir-ing rates to zero) led to increased striatal NoGo activity,as described previously (Frank, 2005a). Second, this ledto increased overall STN and GPi activity, consistent withempirical recordings. Third, DA depletion led to emer-gent network oscillations between the STN, GPi and GPelayers, which have been linked to Parkinson’s tremor asdescribed above. These oscillations were more prominentwhen no motor response was selected (Figure 3d), consis-tent with empirical observations that movements suppressSTN oscillations (Amirnovin et al., 2004), and with thefact that tremor is usually seen in the resting state. Fur-ther, oscillations could be observed even when layer activ-ity levels were averaged across multiple trials (Figure 4a),suggesting that they are highly regular (for a given net-work configuration; random GPe-STN connectivity leadsto variability across networks).Finally, simulated STN le-sions (by removing the STN layer from processing) inDA-depleted networks normalized GPi activity and elim-inated GPi/GPe oscillations (Figure 4b). As mentionedabove, this same pattern of results has been observed as aconsequence of STN lesions in the dopamine-depleted an-imal (Ni et al., 2000). Similarly, because cortex providesthe primary excitatory input onto STN, simulated corticallesions also eliminated oscillations (data not shown), con-sistent with experimental data (Magill, Bolam, & Bevan,2001). In sum, the close correspondence with various ef-fects of DA manipulation on BG firing patterns supportsthe model’s biological plausibility, particularly in light ofthe fact that it was not specifically designed to reproducethese physiological data. Next, the relevance of these pat-terns to response selection processes are considered.

The STN and Action SelectionIf STN lesions improve Parkinson symptoms, it is

natural to consider what deleterious effects they mighthave. In other words, what is the essential computa-tional function of the STN in action selection / decisionmaking? Some evidence comes from the animal litera-ture showing that STN lesions impair response selectionprocesses, and leads to premature responding when hav-ing to suppress competing responses (Baunez & Robbins,1997; Baunez et al., 2001). This leaves open the possibil-ity that the Global NoGo signal provided by the STN isadaptive and allows the animal (or the model) sufficienttime to consider all possible responses before selectingthe most adaptive one. This hypothesis is further sup-ported by observations that low-amplitude STN stimula-tion decreases premature responding in rats (Desbonnet,Temel, Visser-Vandewalle, Blokland, Hornikx, & Stein-busch, 2004). The question is whether a formal simula-tion of STN involvement in BG dynamics can account for

Frank 9

a) b)

C 80% R3B 70% R2A 80% R1

Train

D 70% R4

Test

CDAB R1?

R3? Trained Test (80 vs 70)Condition

50

60

70

80

Accu

racy

Full ModelSTN lesion

Selection TaskTrained and Re-paired

Figure 5: a) Response selection paradigm. Four cues are inde-pendently associated with one of four possible responses. Re-sponses R1 and R3 are reinforced on 80% of trials in the pres-ence of cues A and C, respectively. R2 and R4 are reinforced in70% of trials to cues B and D. The test phase measures the net-work’s ability to choose the 80% over the 70% response whenpresented with cues A and B or C and D together. b) Both in-tact networks and those with STN lesions successfully learnedto choose the appropriate response for each training cue. STNlesions selectively impaired selection among two competing re-sponses, due to premature responding before being able to in-tegrate over all possible responses. Error bars reflect standarderror across 25 runs of the model with random initial synapticweights.

these data in a response selection paradigm.To address this question, a reinforcement learning

paradigm was simulated in which the model was pre-sented with one of four cues, each represented by a col-umn of simultaneously active units in the input layer.The network’s task was to select one of four possible re-sponses for each cue (Figure 5a). “Feedback” is then pro-vided to the network by either increasing or decreasingdopamine levels. The network learns based on the differ-ence in Go/NoGo activity levels in the response selectionand feedback phase, as detailed in Frank (2005) and in theappendix.

The stimulus-response mappings are probabilistic,such that the optimal response for some cues will lead topositive reinforcement (DA bursts) in 80% of trials; in theremaining 20% of trials some alternative response is rein-forced. For all incorrect responses, DA dips are applied.Other cue-response mappings are less reliable, such thatthe optimal response is positively reinforced in only 70%of trials. Networks were trained with 15 epochs consistingof 10 trials of each stimulus cue. As in previous simula-tions, BG networks should be able to learn to select the re-sponse most associated with positive reinforcement basedon Go/NoGo learning within the striatum (Frank, 2005a;Frank et al., 2004). But in these prior simulations, theSTN was not incorporated and was therefore not criticalfor this learning to take place.

To determine whether the STN is beneficial for select-ing among multiple competing responses, a test phase wasadministered. Two cues were presented in the input simul-taneously, one of which had been associated with 80%positive reinforcement if responded to by one response,while the other had been associated with 70% positivereinforcement for an alternative response. Although themodels had not been trained with these stimulus com-binations, they should nevertheless be able to select theresponse that was most likely to result in positive rein-forcement (i.e. the 80% response). However, prematureresponding could result in selection of the 70% reinforcedresponse if its corresponding striatal Go signal happenedto get active prior to that of the 80% response (due to noisein striatum or in the premotor representations themselves).This is precisely this kind of situation that an initial STNGlobal NoGo signal may be useful, so that the networkcan integrate over multiple possible responses before se-lecting the most appropriate one.

Simulations results were consistent with this depic-tion (Figure 5b). While there was no difference betweennetworks in their ability to select the most adaptive re-sponse for each cue, models with STN lesions were im-paired at making high-conflict decisions (e.g., choosingthe best among two positively associated responses). Thisresult is consistent with the notion that the STN is criticalfor preventing premature responding, as networks withoutthe STN were equally likely to choose the 70% responseas the 80% response.3

Further support for the above conclusion comes fromanalysis of model dynamics during the learning and test-ing stages of this task (Figure 6, averaged across trialsof each type). This analysis reveals that the strength ofthe initial STN Global NoGo signal is modulated by thedegree of response conflict present in motor cortex, suchthat if multiple competing responses are active, the net-work may take more time to select a given one. Recallthat in training trials, only one cue was presented that hadbeen most reliably associated with a single response. Thusa minimal amount of response conflict in premotor cortexled to a small initial STN Global NoGo signal (Figure 6a).In these trials, virtually all responses were selected by cy-cle 50 (as evidenced by average Thalamic activity). Incontrast, the STN surge occurred earlier and was larger

3Similar patterns of results were observed in a network that wastrained to select among two responses (instead of four; data not shown).Thus whereas previous models omitting the STN were capable of learn-ing complex probabilistic tasks (Frank, 2005a) and produced the correctpattern of DA-dependent learning biases (in terms of Go vs NoGo striatalrepresentations; Frank et al., 2004), the networks in those studies werenot specifically tested in their ability to select among two responses thathad been independently associated with similar reinforcement values.The present simulations reveal that the STN improves choice selectionin these high-conflict decisions.


a) b)

0 10 20 30 40 50 60 70 80 90 100Cycle

0.00.10.20.30.40.50.60.70.80.91.0

Avg

Activ

ity L

evel

Avg Activity During Selection:Multiple Training Trials

0 10 20 30 40 50 60 70 80 90 100Cycle

0.00.10.20.30.40.50.60.70.80.91.0

Avg

Activ

ity L

evel


Avg Activity During Selection:High-Conflict Test Trials

c) d)

0 10 20 30 40 50 60 70 80 90 100Cycle

0.00.10.20.30.40.50.60.70.80.91.0

Avg

Activ

ity L

evel

STN Lesion:Multiple Training Trials

0 10 20 30 40 50 60 70 80 90 100Cycle

0.00.10.20.30.40.50.60.70.80.91.0

Avg

Activ

ity L

evel


STN Lesion:High-Conflict Test Trials

Figure 6: Average unit activity during response selection as a function of network settling cycles. Data are averaged across unitswithin each area, and across 100 trials of the response selection task depicted in Figure 5. a) Intact network. Dynamics are similarto those in a single trial (Figure 3b), but transitions are less clearcut, because they occur at somewhat different latencies acrossmultiple trials. Nevertheless, virtually all responses during low-conflict training trials were selected by cycle 50, as can be seenby asymptotic Thalamic activity. b) High-conflict test trials (see Figure 5a). Note earlier and larger STN burst due to multipleconflicting cortical responses. Gradual Thalamus activity indicates that the STN Global NoGo signal prevented many responsesfrom being selected until later in network settling. c) In STN lesioned networks, response selection time is similar to that ofintact networks for low-conflict training trials. d) A lack of STN Global NoGo surge in STN-lesioned networks causes prematureresponding in high-conflict trials; all responses are selected between cycles 30 and 50.

Frank 11

in magnitude during high-conflict test trials (Figure 6b),due to increased cortical synaptic activity (from prior cor-tical learning) and resulting in slower response execution(slower increase in average Thalamus activity). Notably,models with simulated STN lesions did not demonstratethis modulation of response time by degree of conflict;these models continued to select all responses by cycle 50in both training and test trials (Figures 6c,d).

Discussion

This work presents a novel computational explorationof the subthalamic nucleus within the overall basal gan-glia circuitry. The model integrates various neural and be-havioral findings and provides insight into the STN rolein response selection and decision making. Consistentwith other BG models (Gurney et al., 2001; Brown et al.,2004), the STN provides a “Global NoGo” signal thatsuppresses all responses. But the current simulations re-vealed that this signal is dynamic, such that it is evokedupon initial response initiation, is then inhibited, and isfinally reactivated during termination, as has been ob-served in various physiological observations (Wichmann,Bergman, & DeLong, 1994; Kolomiets, Deniau, Mailly,Menetrey, Glowinski, & Thierry, 2001; Magill, Sharott,Bevan, Brown, & Bolam, 2004; Maurice et al., 1998a;Bevan, Magill, Terman, Bolam, & Wilson, 2002; Nambuet al., 2002). Further, the degree and duration of STN ac-tivity is directly driven by the amount of response conflictpresent in cortical motor representations, suggesting thatthese STN dynamics are adaptive in preventing prematureresponding for high-conflict decisions.

In the following sections, I discuss how the model cor-responds to various neural and behavioral data in the con-text of response selection and decision making.

Model Correspondence to BG ActivityIn accord with physiological observations (Nambu

et al., 2000; Magill et al., 2004), a burst in STN activityoccurs during the initial stages of response selection. Thisburst is elicited via the “hyperdirect” pathway, as multi-ple competing responses are activated in premotor corti-cal areas. If multiple responses have been associated withadaptive behavior for the particular stimulus context, thenthese will be more active and will therefore more stronglyexcite the STN. The resulting Global NoGo signal delaysresponding until the competition is adequately resolved.When a particular response is facilitated (and the otherssuppressed), this Global NoGo signal is shut off. This oc-curs due to a combination of less overall top-down activ-ity from cortex (since alternative responses are no longeractive), and inhibition of the STN via GPe activity. Sub-sequent striatal NoGo activity can then lead to a second

surge of STN activity (via GPe inhibition), supporting ter-mination of responses. This overall functionality is con-sistent with physiological recordings of the STN duringresponse selection and cortical stimulation. Further, re-cent imaging studies reveal that the human STN is partic-ularly active during high-conflict trials in which prepotentresponses are to be inhibited (Aron & Poldrack, in press).

Relationship to Models of Optimal DecisionMaking

The very same BG dynamics may serve to optimizeresponse times and decision thresholds depending on taskdemands, as has been studied in the context of two-alternative forced choice tasks. Abstract neural and math-ematical models of optimal decision making suggest thatagents must first integrate over processing noise in or-der to extract the best possible decision before makinga response (e.g., Ratcliff, Van Zandt, & McKoon, 1999;Usher & McClelland, 2001; Brown, Gao, Holmes, Bo-gacz, Gilzenrat, & Cohen, 2005). These models makecontact with electrophysiological findings showing thatthe firing rates of “decision” neurons in monkey motorareas gradually increase during forced choice tasks, andwhen these rates cross a decision threshold a choice isexecuted (Gold & Shadlen, 2002; Schall, 2003). Inter-estingly, both animals and humans can dynamically andoptimally adjust the behavioral threshold for when to ex-ecute a response so as to maximize their reward rate, interms of correct responses per unit time (Bogacz et al.,submitted; Simen et al., this issue).

The adaptive and dynamic STN functionality simu-lated here is consistent with the formal requirements de-scribed by the above models. The STN Global NoGo sig-nal effectively achieves this function by allowing both cor-tical and striatal signals to accumulate and compete beforedetermining which response to facilitate. Because STNactivity is dynamically modulated by the level of responseconflict (which itself is determined by prior reward asso-ciations), the STN may contribute to optimally maximiz-ing the response time in a given choice situation such thatfaster responses are achieved for low-conflict decisions,whereas more integration of information can occur forhigh-conflict decisions. Thus, the STN is predicted to playan important role in classical speed-accuracy trade off ef-fects. This account is also consistent with theoretical per-spectives positing that response conflict is represented inrostral anterior cingulate cortex (e.g., Botvinick, Braver,Barch, Carter, & Cohen, 2001; Yeung, Botvinick, & Co-hen, 2004); this area is thought to correspond to the mon-key rostral cingulate motor zone (Picard & Strick, 1996,2001), and may represent the activation of multiple com-peting responses. Notably, this area has direct projectionsto the STN (Orieux, Francois, Feger, & Hirsch, 2002).


Effects of STN Manipulation on Rat Choice Be-havior

The model is also consistent with behavioral find-ings in rats showing that STN lesions worsens, whileSTN stimulation improves, premature responding in high-conflict choice selection paradigms (Baunez & Robbins,1997; Baunez et al., 2001; Desbonnet et al., 2004). Thisis exactly what is predicted by the dynamic Global NoGosignal hypothesized to depend on the STN during re-sponse selection. Nevertheless, a recent finding seem tochallenge this account. Winstanley et al. (2005) foundthat lesions to the STN decreased impulsivity in a delaydiscounting paradigm, such that rats were more likely tomake responses leading to greater long-term rewards in-stead of those leading to short-term reward. However, thepresent model would suggest that rather than the STN le-sion causing enhanced valuation of the delayed reward perse, it may have simply prevented the the rat from reliablychoosing the option that it actually would have wantedto choose (i.e. the immediate reward), if it only couldhave had more time to “consider” both options. Giventhat both choices had some reward value, the lesioned ratsmay have simply been more likely to choose whicheveroption they first considered in any given trial, resultingin relatively more choices for the delayed option. Futureempirical studies are therefore needed to test this account.

Neural Activity During Selection and Oscilla-tions Following DA Depletion

The model also provides evidence for biological plau-sibility at the neural systems level. First, consistentwith neurophysiological observations and as discussedin Frank (2005a), premotor activity is observed withina trial prior to that of the striatum (e.g., Crutcher &Alexander, 1990; Alexander & Crutcher, 1990b, see alsoMink, 1996). This pattern is especially true for well-learned responses, in which premotor cortex can activatethe correct response without requiring striatal facilitation.However, learning-related activity is observed in striatumprior to premotor cortex (Delgado et al., 2005; Pasupa-thy & Miller, 2005) – since initial changes in Go/NoGorepresentations are required before premotor cortex can“stamp-in” the habitual response.

Moreover, in the current simulations, dopamine de-pletion produced emergent oscillatory activity in the STNand BG output nuclei; these oscillations are reliablyobserved in DA-depleted animals and humans and arethought to be the source of Parkinson’s tremor (Bergmanet al., 1994; Bergman et al., 1998; Levy et al., 2000;Raz et al., 2000). Similar oscillations were previouslydescribed and more extensively explored in a biophys-ically detailed conductance model of GPe-STN interac-

tions (Terman, Rubin, Yew, & Wilson, 2002). Althoughthat model did not include a striatum, oscillations wereinduced by applying a constant external inhibitory currentto GPe neurons, so as to simulate enhanced NoGo activityin the DA depleted state. While the current model is notas detailed at the GPe/STN unit level (including just threeionic currents; see appendix), this simpler implementationenabled tractable investigation of systems-level dynamicsamong multiple BG and cortical structures and their rolesin learning and decision making. That we still observeDA-dependent oscillations provides support for the plau-sibility of the approach. Further, oscillatory activity inthe current model was substantially reduced or eliminatedwith simulated lesions to STN or motor cortical areas, ashas also been shown empirically (Bergman et al., 1990;Ni et al., 2000; Magill et al., 2001). Nevertheless, it isacknowledged that there are multiple possible configura-tions that could lead to oscillatory behavior in complex in-teractive circuits such as the BG. Thus the current simula-tions do not prove that they replicate the exact conditionsunder which oscillatory activity and tremor is observed inParkinson’s disease, but merely provide a plausible sce-nario. In particular, it is not currently known whether os-cillations stem from DA depletion to the entire BG, or ifthey would occur with restricted DA depletion within anygiven area (such as the STN or striatum). It is thereforeimportant to be explicit about the cause of oscillations, sothat multiple alternatives could be tested.

In the model, oscillations in various BG nuclei oc-cur via emergent network dynamics following striatal DAdepletion. The lack of D2 receptor stimulation leads tooveractive NoGo units (which are normally inhibited bydopamine), resulting in excessive inhibition of GPe and,in turn, disinhibition of the STN.

Subsequent burst firing in STN propagates back to ex-cite GPe, which now becomes more active and thus be-gins to inhibit STN. The cycle repeats, leading to oscil-lations in STN and GPe. Oscillatory activity in GPi isalso observed, reflecting afferent activity from both STNand GPe. Note we did not need to simulate DA depletionwithin STN, GPe and GPi themselves for oscillations tooccur; instead they arise from DA depletion to the stria-tum. (As in Terman et al. (2002), the strength of GPe ac-tivity is critical for oscillatory behavior; when the strengthof the GPe-STN projection was weakened, oscillationswere no longer observed.)

Therefore, in both models, the critical source of os-cillations is burst firing within STN after disinhibition byGPe. However, STN disinhibition alone is not sufficient toproduce oscillations; additional mechanisms are requiredto generate bursting. In the current model, top-down ex-citatory projections are required to generate burst firingonly in STN units which are are concurrently disinhibited

Frank 13

by GPe and excited by cortex. Alternatively, this burstfiring can occur via intracellular rebound firing mecha-nisms within STN itself Terman et al. (2002). The currentsystems-level account is consistent with data showing thattop-down cortical projections play a key role in the gen-eration of STN bursts and associated oscillations (Magillet al., 2001; Bevan et al., 2002). Moreover, the modelmakes the unique prediction that increased response con-flict in cortex should drive further burst firing in STN un-der DA depletion, and that Parkinson’s tremor should beexacerbated under these conditions. More generally, be-cause cortical regions representing conflict also becomeincreasingly engaged as participants make errors in cog-nitive tasks (e.g, Yeung et al., 2004; Frank et al., 2005),we predict that detectable increases in tremor should beobserved following negative reinforcement.

Model Limitations and Future DirectionsWhile the current model is an advancement over pre-

vious simulations, in that it learns to select among fourcompeting responses (instead of just two) and incorpo-rates the dynamic contribution of the STN, there never-theless remain several important limitations.

Integrating Actor and Critic Functions of the BGIn this and the previous model (Frank, 2005a), we

have not addressed the important question of how re-ward and loss information is computed by systems up-stream of midbrain dopamine neurons; instead we sim-ply assumed this function by externally increasing anddecreasing simulated DA levels during positive reinforce-ment and negative reinforcement, and then examined theireffects on learning and decision making in the BG net-work. Said otherwise, the current work focuses on theactor functions of the BG, and simply assumes the criticfunction. In parallel work, we are investigating how in-teractions between the amygdala and ventral striatal BGregions can support the critic function by learning to as-sociate stimuli with affective states and driving dopamin-ergic firing in the midbrain (O’Reilly, Frank, Hazy, &Watz, submitted; see also Brown et al., 1999; Houk et al.,1995). This work provides a biologically explicit mech-anism for the widely acknowledged relationship betweenthe firing patterns of dopamine neurons during condition-ing paradigms, and those predicted by the abstract math-ematical temporal differences reinforcement learning al-gorithm (Sutton, 1988; Schultz et al., 1997). Preliminarysimulations demonstrate that the current actor BG modelcan learn successfully when dopamine firing is computedby the critic model, rather than applying DA values exter-nally.

The Role of SerotoninAs it stands, the model does not investigate functions

of other neuromodulators beside dopamine, includingserotonin and norepinephrine, both of which are thoughtto play key roles in reinforcement learning and decisionmaking (Daw, Kakade, & Dayan, 2002; Aston-Jones &Cohen, 2005; Harley, 2004). Particularly relevant to thecurrent model, a role for serotonin (5-HT) has been im-plicated in impulsive behavior and response inhibition(Evers et al., 2005; Walderhaug et al., 2002; Winstanleyet al., 2004; but see Clark et al., 2005). It is likely thatthese effects are partially mediated by serotonergic pro-cesses within the STN. Serotonergic neurons of the dor-sal raphe nucleus innervate the STN (Lavoie & Parent,1990), where they are excitatory via densely expressed 5-HT2c receptors (Pompeiano, Palacios, & Mengod, 1994;Xiang, Wang, & Kitai, 2005; Stanford, Kantaria, Cha-hal, Loucif, & Wilson, 2005). In preliminary exploratorysimulations of 5-HT function within the STN of the cur-rent model, background 5-HT levels modulate the gain ofSTN neural activity and enhance Global NoGo signals.Thus, according to the model, serotonin may modulatethe gain of STN Global NoGo signals so as to preventpremature responding and impulsive behavior. Consistentwith this hypothesis, blockade of 5-HT2c receptors leadsto increased premature responding, as well as decreasedlatency to make a correct response, in rats (Winstanley,Theobald, Dalley, Glennon, & Robbins, 2004). Furtherresearch is necessary to determine whether this serotoner-gic effect is mediated selectively in the STN.

Others have emphasized a potential computationalrole for serotonin in negative reinforcement (NoGo) learn-ing, via opponent processes with dopamine (Daw et al.,2002). We have argued that while there is some evi-dence to support this assertion, low levels of DA (or tran-sient DA dips) are still necessary for BG-mediated nega-tive reinforcement learning, and that 5-HT effects may bemore likely mediated in prefrontal cortex rather than BG(e.g., Clarke, Dalley, Crofts, Robbins, & Roberts, 2004;Frank & Claus, in press). According to our modelingframework, 5-HT modulation of prefrontal working mem-ory representations of recent punishments would impactthe extent to which behavioral modifications are madeon a trial-to-trial basis in response to negative feedback.Recent evidence supports this conclusion, showing thatblockade of 5-HT reuptake in humans increases trial-to-trial punishment sensitivity (Chamberlain, Muller, Black-well, Clark, Robbins, & Sahakian, 2006). Furthermore,5-HT could also indirectly affect BG-dependent NoGolearning by lowering DA release, via inhibitory 5-HT re-ceptors onto DA neurons (Nocjar, Roth, & Pehek, 2002).


Frontal Reward Representations in Decision MakingThe current BG model cannot address other more ad-

vanced aspects of human decision making, which wouldrequire taking into account other frontal brain regions. Forexample, in other simulations we have shown that the ven-tromedial and orbitofrontal cortices (OFC) are critical forlearning to make decisions that depend on accurate esti-mates of the expected values of decisions. This is becausein addition to the BG’s specialization in slowly integrat-ing the relative probabilities of reinforcement of alterna-tive decisions, the OFC is specialized to also incorporategraded differences in the relative magnitudes of potentialreinforcement, especially when contingencies change un-expectedly (Frank & Claus, in press). In these cases, ac-tive OFC working memory representations encoding themagnitudes of recent positive and negative reinforcementexperiences may be critical for over-riding the prepotentfrequency associations that the BG system is particularlyspecialized to extract. Future work will examine the roleof medial OFC projections directly to the STN (Mau-rice, Deniau, Glowinski, & Thierry, 1998b). It is pos-sible that in addition to receiving information about thelevel of response conflict from premotor/cingulate areas,the STN may also receive OFC information about the ex-pected magnitude of reward outcomes from each of theseresponse alternatives. This functionality would support aneurobiological mechanism for response threshold adap-tation by reward rate (see Simen, Cohen, & Holmes, thisissue), and could also explain the enhanced choice of de-layed over immediate rewards in STN-lesioned rats (de-scribed above) (Winstanley, Baunez, Theobald, & Rob-bins, 2005). Finally, in addition to these OFC functions,other BG-PFC circuits may be critical for more complexhuman decision making tasks that require explicit plan-ning and computations of if-then scenarios (Houk & Wise,1995; Frank et al., 2001; O’Reilly & Frank, 2006; Hazy,Frank, & O’Reilly, in press).

Conclusion

How do the present simulations provide insight intothe problem of when the subthalamic nucleus is benefi-cial for cognition, compared with situations in which toomuch STN activity may impair cognitive function? A pre-liminary answer to this question may be that the STN isuseful in situations that would otherwise lead to “jump-ing the gun” on decision making processes, by preventingpremature choices. However, when excessive hesitancy isexperienced, the present model would suggest turning offyour STN, especially when adequate information is notavailable to indicate which choice is better. Future com-putational work may help us better understand both thetherapeutic and deleterious effects of STN stimulation on

motor and cognitive processes in Parkinson’s disease andrelated disorders.

Appendix

The model can be obtained by emailing theauthor at [email protected]. For an-imated video captures of model dynamics dur-ing response selection and learning, please seewww.u.arizona.edu/˜mfrank/BGmodel_movies.html

Implementational DetailsThe model is implemented using the Leabra frame-

work (O’Reilly & Munakata, 2000; O’Reilly, 2001).Leabra uses point neurons with excitatory, inhibitory, andleak conductances contributing to an integrated mem-brane potential, which is then thresholded and trans-formed via an x/(x + 1) sigmoidal function to producea rate code output communicated to other units (dis-crete spiking can also be used, but produces noisier re-sults). Each layer uses a k-winners-take-all (kWTA) func-tion that computes an inhibitory conductance that keepsroughly the k most active units above firing threshold andkeeps the rest below threshold.

The membrane potential Vm is updated as a functionof ionic conductances g with reversal (driving) potentialsE as follows:

∆Vm(t) = τ∑

c

gc(t)gc(Ec − Vm(t)) (1)

with 3 channels (c) corresponding to: e excitatory input;l leak current; and i inhibitory input. Following electro-physiological convention, the overall conductance is de-composed into a time-varying component gc(t) computedas a function of the dynamic state of the network, anda constant gc that controls the relative influence of thedifferent conductances. The equilibrium potential can bewritten in a simplified form by setting the excitatory driv-ing potential (Ee) to 1 and the leak and inhibitory drivingpotentials (El and Ei) of 0:

V ∞

m =gege

gege + glgl + gigi(2)

which shows that the neuron is computing a balance be-tween excitation and the opposing forces of leak and in-hibition. This equilibrium form of the equation can beunderstood in terms of a Bayesian decision making frame-work (O’Reilly & Munakata, 2000).

The excitatory net input/conductance ge(t) or ηj iscomputed as the proportion of open excitatory channelsas a function of sending activations times the weight val-

Frank 15

ues:ηj = ge(t) = 〈xiwij〉 =

1

n

∑

i

xiwij (3)

The inhibitory conductance is computed via the kWTAfunction described in the next section, and leak is a con-stant.

Activation communicated to other cells (yj) is athresholded (Θ) sigmoidal function of the membrane po-tential with gain parameter γ:

yj(t) =1

(

1 + 1γ[Vm(t)−Θ]+

) (4)

where [x]+ is a threshold function that returns 0 if x < 0and x if X > 0. Note that if it returns 0, we assumeyj(t) = 0, to avoid dividing by 0. As it is, this functionhas a very sharp threshold, which interferes with gradedlearning learning mechanisms (e.g., gradient descent). Toproduce a less discontinuous deterministic function witha softer threshold, the function is convolved with a Gaus-sian noise kernel (µ = 0, σ = .005), which reflects theintrinsic processing noise of biological neurons:

y∗

j (x) =

∫

∞

−∞

1√2πσ

e−z2/(2σ2)yj(z − x)dz (5)

where x represents the [Vm(t) − Θ]+ value, and y∗

j (x) isthe noise-convolved activation for that value.

Inhibition Within and Between LayersInhibition between layers (i.e for GABAergic projec-

tions between BG layers) is achieved via simple unit in-hibition, where the inhibitory current gi for the unit is de-termind from the net input of the sending unit.

For within layer lateral inhibition (used in Stria-tum and premotor cortex), Leabra uses a kWTA (k-Winners-Take-All) function to achieve inhibitory compe-tition among units within each layer (area). The kWTAfunction computes a uniform level of inhibitory currentfor all units in the layer, such that the k+1th most excitedunit within a layer is generally below its firing threshold,while the kth is typically above threshold. Activation dy-namics similar to those produced by the kWTA functionhave been shown to result from simulated inhibitory in-terneurons that project both feedforward and feedback in-hibition (O’Reilly & Munakata, 2000). Thus, althoughthe kWTA function is somewhat biologically implausiblein its implementation (e.g., requiring global informationabout activation states and using sorting mechanisms), itprovides a computationally effective approximation to bi-ologically plausible inhibitory dynamics.

kWTA is computed via a uniform level of inhibitory

current for all units in the layer as follows:

gi = gΘk+1 + q(gΘ

k − gΘk+1) (6)

where 0 < q < 1 (.25 default used here) is a parameterfor setting the inhibition between the upper bound of gΘ

k

and the lower bound of gΘk+1. These boundary inhibition

values are computed as a function of the level of inhibitionnecessary to keep a unit right at threshold:

gΘi =

g∗e ge(Ee − Θ) + glgl(El − Θ)

Θ − Ei(7)

where g∗

e is the excitatory net input.Two versions of kWTA functions are typically used in

Leabra. In the kWTA function used in the Striatum, gΘk

and gΘk+1 are set to the threshold inhibition value for the

kth and k+1th most excited units, respectively. Thus, theinhibition is placed to allow k units to be above threshold,and the remainder below threshold.

The premotor cortex uses the average-based kWTAversion, gΘ

k is the average gΘi value for the top k most ex-

cited units, and gΘk+1 is the average of gΘ

i for the remain-ing n−k units. This version allows for more flexibility inthe actual number of units active depending on the natureof the activation distribution in the layer and the value ofthe q parameter (which is set to default value of .6). Thisflexibility is necessary for the premotor units to have dif-ferential levels of activity during settling (depending onwhether or not a single response has been facilitated), andalso allows greater activity in high-conflict trials.

LearningSynaptic connection weights were trained using a re-

inforcement learning version of Leabra. The learning al-gorithm involves two phases, and is more biologicallyplausible than standard error backpropagation. In the mi-nus phase, the network settles into activity states based oninput stimuli and its synaptic weights, ultimately “choos-ing” a response. In the plus phase, the network reset-tles in the same manner, with the only difference beinga change in simulated dopamine: an increase of SNc unitfiring from 0.5 to 1.0 for correct responses, and a decreaseto zero SNc firing for incorrect responses (Frank, 2005a).Connection weights are adjusted to learn on the differencebetween pre and postsynaptic activation product acrossthe minus and plus phases (O’Reilly, 1996).

References

Albin, R. L., Young, A. B., & Penney, J. B. (1989). Thefunctional anatomy of basal ganglia disorders. Trendsin Neurosciences, 12, 366–375.


Param Value Param Value Param Value Param Value Param Value Param ValueEl 0.15 gl 0.10 Ei 0.15 gi 1.0 Ee 1.00 ge 1.0Vrest 0.15 Θ 0.25 γ 600 khebb .01 ε .001Striatum (k=4) gl 1.0* Θ, +DA 0.32* γ 2500* γ, +DA 10000* γ, -DA 300*GPi El 0.28* gl 3.0* Vrest 0.26*GPe El 0.26* gl 1.0* Vrest 0.26*STN El 0.2* gl 1.0* Vrest 0.25*Thal gi 1.7* ge 0.5*Premotor (k=3) ε 1e-5* khebb 1* Vm noise µ =.0015 Vm noise σ=.0015

Table 1: BG model parameters. First two rows indicate standard default parameters used in 100’s of simulations with Leabra software; theseparameters are used in the model except where noted with an * for specialized functions of the BG layers. Striatal units have a higher firing thresholdθ and higher gain γ during DA bursts (“+DA”), and lower γ during DA dips, to simulate contrast enhancement and reduction (Frank, 2005). GPand STN units have higher than normal El, gl and Vrest, leading to tonic baseline activity in the absence of synaptic input. Thal units have highgi and low ge, enabling a default strong inhibition from BG output and only allowing top-down excitatory activity if disinhibited, thereby serving agating function. Premotor units have Gaussian noise added to the membrane potential, learn with a slow learning rate via purely Hebbian learning.k (kWTA) parameters are shown for striatum and premotor areas, which have within-layer lateral inhibition.

Alexander, G. E., & Crutcher, M. D. (1990a). Functionalarchitecture of basal ganglia cicuits: neural substratesof parallel processing. Trends in Neuroscience, 13,266–271.

Alexander, G. E., & Crutcher, M. D. (1990b). Preparationfor movement: Neural representations of intended di-rection in three motor areas of the monkey. Journal ofNeurophysiology, 64, 133–50.

Alexander, G. E., DeLong, M. R., & Strick, P. L. (1986).Parallel organization of functionally segregated circuitslinking basal ganglia and cortex. Annual Review ofNeuroscience, 9, 357–381.

Amirnovin, R., Williams, Z. M., Cosgrove, G. R., & Es-kandar, E. N. (2004). Visually guided movements sup-press subthalamic oscillations in parkinson’s diseasepatients. Journal of Neuroscience, 24(50), 11302–11306.

Aron, A. R., & Poldrack, R. A. (in press). Cortical andsubcortical contributions to stop signal response inhi-bition: role of the subthalamic nucleus. Journal ofNeuroscience.

Ashby, F. G., Noble, S., Ell, S. W., Filoteo, J. V., &Waldron, E. M. (2003). Category learning deficits inParkinson’s Disease. Neuropsychology, 17, 133–42.

Aston-Jones, G., & Cohen, J. D. (2005). An integra-tive theory of locus coeruleus-norepinephrine function:Adaptive gain and optimal performance. Annu. Rev.Neurosci., 28, 403–450.

Aubert, I., Ghorayeb, I., Normand, E., & Bloch, B.(2000). Phenotypical characterization of the neuronsexpressing the D1 and D2 dopamine receptors in themonkey striatum. Journal of Comparative Neurology,418, 22–32.

Bar-Gad, I., Morris, G., & Bergman, H. (2003). Infor-mation processing, dimensionality reduction and rein-forcement learning in the basal ganglia. Progress inNeurobiology, 71, 439–73.

Basso, M. A., & Wurtz, R. H. (2002). Neuronal activity insubstantia nigra pars reticulata during target selection.J Neurosci, 22(5), 1883–1894.

Baunez, C., Humby, T., Eagle, D. M., Ryan, L. J., Dun-nett, S. B., & Robbins, T. W. (2001). Effects of STN le-sions on simple vs choice reaction time tasks in the rat:Preserved motor readiness, but impaired response se-lection. European Journal of Neuroscience, 13, 1609–16.

Baunez, C., & Robbins, T. W. (1997). Bilateral lesionsof the subthalamic nucleus induce multiple deficits inan attentional task in rats. European Journal of Neuro-science, 9(10), 2086–2099.

Bayer, H. M., & Glimcher, P. W. (2005). Midbraindopamine neurons encode a quantitative reward predic-tion error signal. Neuron, 47(1), 129–141.

Beiser, D. G., & Houk, J. C. (1998). Model of cortical-basal ganglionic processing: encoding the serial orderof sensory events. Journal of Neurophysiology, 79,3168–3188.

Benazzouz, A., & Hallett, M. (2000). Mechanism of ac-tion of deep brain stimulation. Neurology, 55, S13–S16.

Bergman, H., Feingold, A., Nini, A., Raz, A., Slovin, H.,Abeles, M., & Vaadia, E. (1998). Physiological as-pects of information processing in the basal ganglia ofnormal and parkinsonian primates. Trends in Neuro-sciences, 21, 32–8.

Bergman, H., Wichmann, T., & DeLong, M. R. (1990).Reversal of experimental parkinsonism by lesions ofthe subthalamic nucleus. Science, 249, 1436–8.

Bergman, H., Wichmann, T., Karmon, B., & DeLong,M. R. (1994). The primate subthalamic nucleus. II.neuronal activity in the MPTP model of parkinsonism.Journal of Neurophysiology, 72, 507–20.

Frank 17

Bevan, M. D., Magill, P. J., Terman, D., Bolam, J. P., &Wilson, C. J. (2002). Move to the rhythm: oscilla-tions in the subthalamic nucleus-external globus pal-lidus network. Trends in Neurosciences, 25, 525–531.

Bogacz, R., Brown, E., Moehlis, J., Hu, P., Holmes, P.,& Cohen, J. D. (submitted). The physics of optimaldecision making: A formal analysis of models of per-formance in two-alternative forced choice tasks.

Bokura, H., Yamaguchi, S., & Kobayashi, S. (2005).Event-related potentials for response inhibition inParkinson’s disease. Neuropsychologia, 43, 967–975.

Boraud, T., Bezard, E., Bioulac, B., & Gross, E. (2002).From single extracellular unit recording in experimen-tal and human Parkinsonism to the development of afunctional concept of the role played by the basal gan-glia in motor control. Progress in Neurobiology, 66(4),265–83.

Botvinick, M. M., Braver, T. S., Barch, D. M., Carter,C. S., & Cohen, J. D. (2001). Conflict monitoring andcognitive control. Psychological Review, 108, 624–652.

Brown, E., Gao, J., Holmes, P., Bogacz, R., Gilzenrat, M.,& Cohen, J. D. (2005). Simple neural networks that op-timize decisions. International Journal of Bifurcationand Chaos, 15, 803–826.

Brown, J., Bullock, D., & Grossberg, S. (1999). Howthe basal ganglia use parallel excitatory and inhibitorylearning pathways to selectively respond to unexpectedrewarding cues. Journal of Neuroscience, 19, 10502–10511.

Brown, J. W., Bullock, D., & Grossberg, S. (2004). Howlaminar frontal cortex and basal ganglia circuits inter-act to control planned and reactive saccades. NeuralNetworks, 17, 471–510.

Brown, R. G., & Marsden, C. D. (1988). Internal versusexternal cues and the control of attention in Parkinson’sdisease. Brain, 111, 323–345.

Chamberlain, S. R., Muller, U., Blackwell, A. D., Clark,L., Robbins, T. W., & Sahakian, B. J. (2006). Neuro-chemical Modulation of Response Inhibition and Prob-abilistic Learning in Humans. Science, 311(5762),861–863.

Charbonneau, D., Riopelle, R. J., & Beninger, R. J.(1996). Impaired incentive learning in treated Parkin-son’s disease. Can J Neurol Sci, 23(4), 271–8.

Choi, W. Y., Balsam, P. D., & Horvitz, J. C. (2005). Ex-tended habit training reduces dopamine mediation ofappetitive response expression. J Neurosci, 25(29),6729–6733.

Clark, L., Roiser, J. P., Cools, R., Rubinsztein, D. C., Sa-hakian, B. J., & Robbins, T. W. (2005). Stop signal

response inhibition is not modulated by tryptophan de-pletion or the serotonin transporter polymorphism inhealthy volunteers: implications for the 5-HT theory ofimpulsivity. Psychopharmacology (Berl), 1–9. JOUR-NAL ARTICLE.

Clarke, H. F., Dalley, J. W., Crofts, H. S., Robbins, T. W.,& Roberts, A. C. (2004). Cognitive inflexibility af-ter prefrontal serotonin depletion. Science, 304(5672),878–880.

Cools, R. (2005). Dopaminergic modulation of cogni-tive function – implications for L-DOPA treatment inParkinson’s disease. Neuroscience and BiobehavioralReviews, 1–23.

Cools, R., Barker, R. A., Sahakian, B. J., & Robbins,T. W. (2001a). Enhanced or impaired cognitive func-tion in Parkinson’s disease as a function of dopamin-ergic medication and task demands. Cerebral Cortex,11, 1136–1143.

Cools, R., Barker, R. A., Sahakian, B. J., & Robbins,T. W. (2001b). Mechanisms of cognitive set flexibil-ity in parkinson’s disease. Brain, 124, 2503–2512.

Cools, R., Barker, R. A., Sahakian, B. J., & Robbins, T. W.(2003a). L-Dopa medication remediates cognitive in-flexibility, but increases impulsivity in patients withParkinson’s disease. Neuropsychologia, 41, 1431–1441.

Cools, R., Barker, R. A., Sahakian, B. J., & Robbins, T. W.(2003b). L-Dopa medication remediates cognitive in-flexibility, but increases impulsivity in patients withParkinson’s disease. Neuropsychologia, 41, 1431–1441.

Crutcher, M. D., & Alexander, G. E. (1990). Movement-related neuronal activity selectively coding either di-rection or muscle pattern in three motor areas of themonkey. Journal of Neurophysiology, 64, 151–163.

Czernecki, V., Pillon, B., Houeto, J. L., Pochon, J. B.,Levy, R., & Dubois, B. (2002). Motivation, reward,and parkinson‘s disease: influence of dopatherapy.Neuropsychologia, 40, 2257–2267.

Daw, N. D., Kakade, S., & Dayan, P. (2002). Opponentinteractions between serotonin and dopamine. NeuralNetworks, 15, 603–616.

Delgado, M. R., Locke, H. M., Stenger, V. A., & Fiez,J. A. (2003). Dorsal striatum responses to reward andpunishment: Effects of valence and magnitude manip-ulations. Cognitive, Affective, and Behavioral Neuro-science, 3, 27–38.

Delgado, M. R., Miller, M. M., Inati, S., & Phelps, E. A.(2005). An fMRI study of reward-related probabilitylearning. Neuroimage, 24, 862–73.


DeLong, M. R. (1990). Primate models of movementdisorders of basal ganglia origin. Trends in Neuro-sciences, 13, 281–5.

Desbonnet, L., Temel, Y., Visser-Vandewalle, V., Blok-land, A., Hornikx, V., & Steinbusch, H. W. (2004).Premature responding following bilateral stimulationof the rat subthalamic nucleus is amplitude and fre-quency dependent. Brain Research, 1008, 198–204.

Evers, E. A., Cools, R., Clark, L., van der Veen, F. M.,Jolles, J., Sahakian, B., & Robbins, T. W. (2005). Sero-tonergic modulation of prefrontal cortex during nega-tive feedback in probabilistic reversal learning. Neu-ropsychopharmacology.

Frank, M. J. (2005a). Dynamic dopamine modulationin the basal ganglia: A neurocomputational accountof cognitive deficits in medicated and non-medicatedParkinsonism. Journal of Cognitive Neuroscience, 17,51–72.

Frank, M. J. (2005b). When and when not to use yoursubthalamic nucleus: Lessons from a computationalmodel of the basal ganglia. Modelling Natural ActionSelection: Proceedings of an International Workshop,53–60.

Frank, M. J., & Claus, E. D. (in press). Anatomy of a de-cision: Striato-orbitofrontal interactions in reinforce-ment learning, decision making and reversal. Psycho-logical Review.

Frank, M. J., Loughry, B., & O’Reilly, R. C. (2001). Inter-actions between the frontal cortex and basal ganglia inworking memory: A computational model. Cognitive,Affective, and Behavioral Neuroscience, 1, 137–160.

Frank, M. J., & O’Reilly, R. C. (in press). A mechanisticaccount of striatal dopamine function in human cogni-tion: Psychopharmacological studies with cabergolineand haloperidol. Behavioral Neuroscience.

Frank, M. J., OReilly, R. C., & Curran, T. (in press). Whenmemory fails, intuition reigns: Midazolam enhancesimplicit inference in humans. Psychological Science.

Frank, M. J., Seeberger, L. C., & O’Reilly, R. C. (2004).By carrot or by stick: Cognitive reinforcement learningin Parkinsonism. Science, 306, 1940–3.

Frank, M. J., Woroch, B. S., & Curran, T. (2005). Error-related negativity predicts reinforcement learning andconflict biases. Neuron, 47, 495–501.

Gerfen, C. R. (1992). The neostriatal mosaic: multiplelevels of compartmental organization in the basal gan-glia. Annual Review of Neuroscience, 15, 285–320.

Gerfen, C. R. (2000). Molecular effects of dopamine onstriatal projection pathways. Trends in Neurosciences,23, S64–S70.

Gerfen, C. R., Keefe, K. A., & Gauda, E. B. (1995).D1 and D2 dopamine receptor function in the stria-tum: Coactivation of D1- and D2-dopamine receptorson separate populations of neurons results in potenti-ated immediate early gene response in D1-containingneurons. Journal of Neuroscience, 15, 8167–8176.

Gerfen, C. R., & Wilson, C. (1996). The basal ganglia. InL. Swanson, A. Bjorkland, & T. Hokfelt (Eds.), Hand-book of chemical neuroanatomy. vol 12: Integratedsystems of the CNS. (pp. 371–468). Amsterdam: El-sevier.

Gold, J. I., & Shadlen, M. N. (2002). Banburismus andthe brain: Decoding the relationship between sensorystimuli, decisions, and reward. Neuron, 36, 299–308.

Gotham, A. M., Brown, R. G., & Marsden, C. D. (1988).’Frontal’ cognitive function in patients with Parkin-son’s disease ’on’ and ’off’ levodopa. Brain, 111, 299–321.

Gurney, K., Prescott, T. J., & Redgrave, P. (2001). Acomputational model of action selection in the basalganglia. I. a new functional anatomy. Biological Cy-bernetics, 84, 401–410.

Gurney, K. N., Humphries, M., Wood, R., Prescott, T. J.,& Redgrave, P. (2004). Testing computational hy-potheses of brain systems function: a case study withthe basal ganglia. Network, 15(4), 263–290.

Harley, T. (2004). Does cognitive neuropsychology havea future? Cognitive Neuropsychology, 21(1), 3–16.

Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (in press).Banishing the homunculus: Making working memorywork. Neuroscience.

Hebb, D. O. (1949). The organization of behavior. NewYork: Wiley.

Hernandez-Lopez, S., Bargas, J., Surmeier, D. J., Reyes,A., & Galarraga, E. (1997). D1 receptor activation en-hances evoked discharge in neostriatal medium spinyneurons by modulating an L-type Ca2+ conductance.Journal of Neuroscience, 17, 3334–42.

Hernandez-Lopez, S., Tkatch, T., Perez-Garci, E., Galar-raga, E., Bargas, J., Hamm, H., & Surmeier, D. J.(2000). D2 dopamine receptors in striatal mediumspiny neurons reduce L-type Ca2+ currents and ex-citability via a novel PLCβ1-IP3-calcineurin-signalingcascade. Journal of Neuroscience, 20, 8987–95.

Hikosaka, O. (1994). Role of basal ganglia in controlof innate movements, learned behaviour and cognition.In G. Percheron, J. McKenzie, & J. Feger (Eds.), Thebasal ganglia iv: new ideas and data on structure andfunction. (pp. 598–96). Plenum.

Holroyd, C. B., & Coles, M. G. H. (2002). The neural ba-sis of human error processing: Reinforcement learning,

Frank 19

dopamine, and the error-related negativity. Psycholog-ical Review, 109, 679–709.

Houk, J. C. (2005). Agents of the mind. Biol Cybern,92(6), 427–437.

Houk, J. C., Adams, J. L., & Barto, A. G. (1995). A modelof how the basal ganglia generate and use neural sig-nals that predict reinforcement. In J. C. Houk, J. L.Davis, & D. G. Beiser (Eds.), Models of informationprocessing in the basal ganglia (pp. 233–248). Cam-bridge, MA: MIT Press.

Houk, J. C., & Wise, S. P. (1995). Distributed modu-lar architectures linking basal ganglia, cerebellum, andcerebral cortex: their role in planning and controllingaction. Cerebral Cortex, 5, 95–110.

Jackson, G. M., Jackson, S., Harrison, J., Henderson,L., & Kennard, C. (1995). Serial reaction time learn-ing and Parkinson’s disease: evidence for a procedurallearning deficit. Neuropsychologia, 33, 577–593.

Jiang, H., Stein, B. E., & McHaffie, J. G. (2003). Oppos-ing basal ganglia processes shape midbrain visuomotoractivity bilaterally. Nature, 419, 982–985.

Joel, D., & Weiner, I. (1999). Striatal contention schedul-ing and the split circuit scheme of basal ganglia-thalamocortical circuitry: From anatomy to behaviour.In R. Miller, & J. R. Wickens (Eds.), Conceptual ad-vances in brain research: Brain dynamics and the stri-atal complex. (pp. 209–36). Harwood Academic Pub-lishers.

Karachi, C., Yelnik, J., Tande, D., Tremblay, L., Hirsch,E. C., & Francois, C. (2004). The pallidosubthala-mic projection: An anatomical substrate for nonmo-tor functions of the subthalamic nucleus in primates.Movement Disorders.

Kawaguchi, Y., Wilson, C. J., & Emson, P. C. (1990).Projection subtypes of rat neostriatal matrix cells re-vealed by intracellular injection of biocytin. J Neu-rosci, 10(10), 3421–3438.

Kish, S. J., Shannak, K., & Hornykiewicz, O. (1988). Un-even pattern of dopamine loss in the striatum of pa-tients with idiopathic Parkinson’s disease. New Eng-land Journal of Medecine, 318, 876–880.

Knowlton, B. J., Mangels, J. A., & Squire, L. R. (1996).a neostriatal habit learning system in humans. Science,273, 1399.

Kolomiets, B. P., Deniau, J. M., Mailly, P., Menetrey,A., Glowinski, J., & Thierry, A. M. (2001). Segre-gation and convergence of information flow throughthe cortico-subthalamic pathways. J Neurosci, 21(15),5764–5772.

Lavoie, B., & Parent, A. (1990). Immunohistochemi-cal study of the serotoninergic innervation of the basal

ganglia in the squirrel monkey. J Comp Neurol, 299(1),1–16.

Lei, W., Jiao, Y., Del Mar, N., & Reiner, A. (2004). Ev-idence for differential cortical input to direct pathwayversus indirect pathway striatal projection neurons inrats. J Neurosci, 24(38), 8289–8299.

Levesque, M., & Parent, A. (2005). The striatofugal fibersystem in primates: A reevaluation of its organizationbased on single-axon tracing studies. Proc Natl AcadSci U S A.

Levy, R., Hutchison, W. D., Lozano, A. M., & Dostro-vsky, J. O. (2000). High-frequency synchronization ofneuronal activity in the subthalamic nucleus of parkin-sonian patients with limb tremor. Journal of Neuro-science, 20, 7766.

Maddox, W. T., & Filoteo, J. V. (2001). Striatal contri-butions to category learning: Quantitative modeling ofsimple linear and complex non-linear rule learning inpatients with Parkinson’s Disease. Journal of the In-ternational Neuropsychological Society, 7, 710–727.

Magill, P. J., Bolam, J. P., & Bevan, M. D. (2001).Dopamine regulates the impact of the cerebral cortexon the subthalamic nucleus-globus pallidus network.Neuroscience, 106, 313–330.

Magill, P. J., Sharott, A., Bevan, M. D., Brown, P., & Bo-lam, J. P. (2004). Synchronous unit activity and lo-cal field potentials evoked in the subthalamic nucleusby cortical stimulation. Journal of Neurophysiology,92(2), 700–714.

Mahon, S., Casassus, G., Mulle, C., & Charpier, S.(2003). Spike-dependent intrinsic plasticity increasesfiring probability in rat striatal neurons in vivo. J Phys-iol, 550(Pt 3), 947–959.

Maurice, N., Deniau, J.-M., Glowinski, J., & Thierry, A.-M. (1998a). Relationships between the prefrontal cor-tex and the basal ganglia in the rat: Physiology of thecorticosubthalamic circuits. Journal of Neuroscience,18, 9539.

Maurice, N., Deniau, J.-M., Glowinski, J., & Thierry, A.-M. (1998b). Relationships between the prefrontal cor-tex and the basal ganglia in the rat: Physiology of thecorticosubthalamic circuits. Journal of Neuroscience,18, 9539.

McAuley, J. H. (2003). The physiological basis of clinicaldeficits in Parkinson’s disease. Progress in Neurobiol-ogy, 69, 27–48.

Mehta, M. A., Swainson, R., Ogilvie, A. D., Sahakian,B. J., & Robbins, T. W. (2001). Improved short-termspatial memory but impaired reversal learning follow-ing the dopamine D2 agonist bromocriptine in humanvolunteers. Psychopharmacology, 159, 10–20.


Meissner, W., Leblois, A., Hansel, D., Bioulac, B., Gross,C. E., Benazzouz, A., & Boraud, T. (2005). Subthala-mic high frequency stimulation resets subthalamic fir-ing and reduces abnormal oscillations. Brain, 128(Pt10), 2372–2382.

Middleton, F. A., & Strick, P. L. (2000). Basal gangliaoutput and cognition: Evidence from anatomical, be-havioral, and clinical studies. Brain and Cognition, 42,183–200.

Middleton, F. A., & Strick, P. L. (2002). Basal-ganglia‘projections’ to the prefrontal cortex of the primate.Cerebral Cortex, 12, 926–935.

Miller, W., & Delong, M. R. (1987). Altered tonic ac-tivity of neurons in the globus pallidus and subthala-mic nucleus in the primate MPTP model of parkin-sonism. In M. B. Carpenter, & A. Jayaraman (Eds.),The basal ganglia, Vol. II (pp. 415–427). New York:Plenum Press.

Mink, J. W. (1996). The basal ganglia: Focused selectionand inhibition of competing motor programs. Progressin Neurobiology, 50, 381–425.

Nambu, A., Kaneda, K., Tokuno, H., & Takada, M.(2002). Organization of corticostriatal motor inputsin monkey putamen. Journal of Neurophysiology, 88,1830–1842.

Nambu, A., Tokuno, H., Hamada, I., Kita, H., Imanishi,M., Akazawa, T., Ikeuchi, Y., & Hasegawa, N. (2000).Excitatory cortical inputs to pallidal neurons via thesubthalamic nucleus in the monkey. Journal of Neu-rophysiology, 84, 289–300.

Ni, Z., Bouali-Benazzouz, R., Gao, D., Benabid, A., &Benazzouz, A. (2000). Changes in the firing pattern ofglobus pallidus neurons after the degeneration of ni-grostriatal pathway are mediated by the subthalamicnucleus in rat. European Journal of Neuroscience, 12,4338–44.

Nocjar, C., Roth, B. L., & Pehek, E. A. (2002). Localiza-tion of 5-HT(2A) receptors on dopamine cells in sub-nuclei of the midbrain A10 cell group. Neuroscience,111, 163–76.

O’Reilly, R. C. (1996). Biologically plausible error-drivenlearning using local activation differences: The gener-alized recirculation algorithm. Neural Computation,8(5), 895–938.

O’Reilly, R. C. (2001). Generalization in interactive net-works: The benefits of inhibitory competition and Heb-bian learning. Neural Computation, 13, 1199–1242.

O’Reilly, R. C., & Frank, M. J. (2006). Making workingmemory work: A computational model of learning inthe prefrontal cortex and basal ganglia. Neural Com-putation, 18, 283–328.

O’Reilly, R. C., Frank, M. J., Hazy, T. E., & Watz, B. (sub-mitted). Rewards are timeless: The primary value andlearned value (PVLV) Pavlovian learning algorithm.

O’Reilly, R. C., & Munakata, Y. (2000). Computationalexplorations in cognitive neuroscience: Understandingthe mind by simulating the brain. Cambridge, MA:MIT Press.

Orieux, G., Francois, C., Feger, J., & Hirsch, E. C.(2002). Consequences of dopaminergic denervation onthe metabolic activity of the cortical neurons project-ing to the subthalamic nucleus in the rat. Journal ofNeuroscience, 22, 8762–8770.

Pan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I.(2005). Dopamine cells respond to predicted eventsduring classical conditioning: evidence for eligibilitytraces in the reward-learning network. J Neurosci,25(26), 6235–6242.

Parent, A., & Hazrati, L. (1995). Functional anatomy ofthe basal ganglia. II. the place of subthalamic nucleusand external pallidum in basal ganglia circuitry. BrainResearch Reviews, 20, 128–54.

Parkinson, J. A., Dalley, J. W., Cardinal, R. N., Bam-ford, A., Fehnert, B., Lachenal, G., Rudarakanchana,N., Halkerston, K. M., Robbins, T. W., & Everitt,B. J. (2002). Nucleus accumbens dopamine depletionimpairs both acquisition and performance of appeti-tive Pavlovian approach behaviour: implications formesoaccumbens dopamine function. Behavioral BrainResearch, 137, 149–63.

Pasupathy, A., & Miller, E. K. (2005). Different timecourses for learning-related activity in the prefrontalcortex and striatum. Nature, 433, 873–876.

Picard, N., & Strick, P. L. (1996). Motor areas of themedial wall: A review of their location and functionalactivation. Cerebral Cortex, 6(3), 342–353.

Picard, N., & Strick, P. L. (2001). Imaging the premotorareas. Curr Opin Neurobiol, 11(6), 663–672.

Pompeiano, M., Palacios, J. M., & Mengod, G. (1994).Distribution of the serotonin 5-HT2 receptor familymRNAs: comparison between 5-HT2A and 5-HT2Creceptors. Brain Res Mol Brain Res, 23(1-2), 163–178.

Pothos, E. N., Davila, V., & Sulzer, D. (1998). Presynapticrecording of quanta from midbrain dopamine neuronsand modulation of the quantal size. J Neurosci, 18(11),4106–18.

Ratcliff, R., Van Zandt, T., & McKoon, G. (1999). Con-nectionist and diffusion models of reaction time. Psy-chological Review, 106, 261.

Raz, A., Vaadia, E., & Bergman, H. (2000). Firing pat-terns and correlations of spontaneous discharge of pall-

Frank 21

idal neurons in the normal and the tremulous 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine vervet model ofparkinsonism. Journal of Neuroscience, 20, 8559–71.

Redgrave, P., Prescott, T. J., & Gurney, K. (1999). Thebasal ganglia: a vertebrate solution to the selectionproblem? Neuroscience, 89, 1009.

Ridley, R. M., Haystead, T. A., & Baker, H. F. (1981). Ananalysis of visual object reversal learning in the mar-moset after amphetamine and haloperidol. PharmacolBiochem Behav, 14(3), 345–51.

Robertson, G. S., Vincent, S. R., & Fibiger, H. C. (1992).D1 and D2 dopamine receptors differentially regulatec-fos expression in striatonigral and stiratopallidal neu-rons. Neuroscience, 49, 285–96.

Rogers, R. D., Sahakian, B. J., Hodges, J. R., Polkey,C. E., Kennard, C., & Robbins, T. W. (1998). Dissoci-ating executive mechanisms of task control followingfrontal damage and Parkinson’s disease. Brain, 121,815–842.

Rubchinsky, L. L., Kopell, N., & Sigvardt, K. A. (2003).Modeling facilitation and inhibition of competing mo-tor programs in basal ganglia subthalamic nucleus-pallidal circuits. Proceedings of the National Academyof Sciences, 100, 14427–32.

Salin, P., Hajji, M. D., & Kerkerian-Le Goff, L. (1996).Bilateral 6-hydroxydopamine-induced lesion of the ni-grostriatal dopamine pathway reproduces the effects ofunilateral lesion on substance P but not on enkephalinexpression in rat basal ganglia. European Journal ofNeuroscience, 8, 1746–1757.

Sato, F., Parent, M., Levesque, M., & Parent, A. (2000).Axonal branching pattern of neurons of the subthala-mic nucleus in primates. Journal of Comparative Neu-rology, 424, 142–52.

Satoh, T., Nakai, S., Sato, T., & Kimura, M. (2003). Cor-related coding of motivation and outcome of decisionby dopamine neurons. Journal of Neuroscience, 23,9913–9923.

Schall, J. D. (2003). Neural correlates of decision pro-cesses: neural and mental chronometry. Current Opin-ion in Neurobiology, 13, 182–186.

Schultz, W. (2002). Getting formal with dopamine andreward. Neuron, 36, 241–263.

Schultz, W., Dayan, P., & Montague, P. R. (1997). A neu-ral substrate of prediction and reward. Science, 275,1593.

Shohamy, D., Myers, C. E., Geghman, K. D., Sage, J.,& Gluck, M. A. (in press). L-Dopa impairs learning,but spares generalization, in Parkinson’s disease. Neu-ropsychologia.

Shohamy, D., Myers, C. E., Grossman, S., Sage, J., &Gluck, M. A. (2005a). The role of dopamine in cogni-tive sequence learning: evidence from Parkinson’s dis-ease. Behav Brain Res, 156(2), 191–9.

Shohamy, D., Myers, C. E., Grossman, S., Sage, J., &Gluck, M. A. (2005b). The role of dopamine in cogni-tive sequence learning: evidence from Parkinson’s dis-ease. Behav Brain Res, 156(2), 191–9.

Simen, P., Cohen, J., & Holmes, P. (this issue). Rapid de-cision threshold modulation by reward rate in a neuralnetwork. Neural Networks.

Smith, A. G., Neill, J. C., & Costall, B. (1999). Thedopamine D2/D3 receptor agonist 7-OH-DPAT in-duces cognitive impairment in the marmoset. Pharma-cology, Biochemistry and Behavior, 63(2), 201–211.

Smith-Roe, S. L., & Kelley, A. E. (2000). Coincident ac-tivation of NMDA and dopamine D1 receptors withinthe nucleus accumbens core is required for appetitiveinstrumental learning. Journal of Neuroscience, 22,7737–42.

Stanford, I. M., Kantaria, M. A., Chahal, H. S., Loucif,K. C., & Wilson, C. L. (2005). 5-Hydroxytryptamineinduced excitation and inhibition in the subthalamicnucleus: Action at 5-HT(2C), 5-HT(4) and 5-HT(1A)receptors. Neuropharmacology. JOURNAL ARTI-CLE.

Sutton, R. S. (1988). Learning to predict by the methodof temporal diferences. Machine Learning, 3, 9–44.

Swainson, R., Rogers, R. D., Sahakian, B. J., Summers,B. A., Polkey, C. E., & Robbins, T. W. (2000). Prob-abilistic learning and reversal deficits in patients withParkinson’s disease or frontal or temporal lobe lesions:Possible adverse effects of dopaminergic medication.Neuropsychologia, 38, 596.

Terman, D., Rubin, J. E., Yew, A. C., & Wilson, C. J.(2002). Activity patterns in a model for the subtha-lamopallidal network of the basal ganglia. Journal ofNeuroscience, 22, 2963–2976.

Usher, M., & McClelland, J. L. (2001). The time course ofperceptual choice: The leaky, competing accumulatormodel. Psychological Review, 108, 550–592.

Walderhaug, E., Lunde, H., Nordvik, J. E., Landro,N. I. Refsum, H., & Magnusson, A. (2002). Loweringof serotonin by rapid tryptophan depletion increasesimpulsiveness in normal individuals. Psychopharma-cology, 164, 385–91.

Wichmann, T., Bergman, H., & DeLong, M. R. (1994).The primate subthalamic nucleus. I. functional proper-ties in intact animals. Journal of Neurophysiology, 72,494–506.


Winstanley, C. A., Baunez, C., Theobald, D. E. H., &Robbins, T. W. (2005). Lesions to the subthalamic nu-cleus decrease impulsive choice but impair autoshap-ing in rats: the importance of the basal ganglia inPavlovian conditioning and impulse control. EuropeanJournal of Neuroscience, 21(11), 3107–3116.

Winstanley, C. A., Theobald, D. E. H., Dalley, J. W., Glen-non, J. C., & Robbins, T. W. (2004). 5-HT2A and 5-HT2C receptor antagonists have opposing effects ona measure of impulsivity: interactions with global 5-HT depletion. Psychopharmacology (Berl), 176(3-4),376–385.

Witt, K., Pulkowski, U., Herzog, J., Lorenz, D., Hamel,W., Deuschl, G., & Krack, P. (2004). Deep brain stim-ulation of the subthalamic nucleus improves cognitiveflexibility but impairs response inhibition in Parkin-son’s disease. Archives of Neurology, 61, 697–700.

Wu, Y., Richard, S., & Parent, A. (2000). The organiza-tion of the striatal output system: a single-cell juxta-cellular labeling study in the rat. Neurosci Res, 38(1),49–62.

Xiang, Z., Wang, L., & Kitai, S. T. (2005). Modulation ofspontaneous firing in rat subthalamic neurons by 5-htreceptor subtypes. Journal of Neurophysiology, 93(3),1145–1157.

Yeung, N., Botvinick, M. M., & Cohen, J. D. (2004).The neural basis of error detection: Conflict monitor-ing and the error-related negativity. Psychological Re-view, 111(4), 931–959.

hold your horses: a dynamic computational role for the...

Documents