Behavioural Brain Research 199 (2009) 43–52

Contents lists available at ScienceDirect

Behavioural Brain Research

journal homepage: www.elsevier.com/locate/bbr

Review

The integrative function of the basal ganglia in instrumental conditioning夽

Bernard W. Balleine ∗ , Mimi Liljeholm, Sean B. Ostlund
Department of Psychology and the Brain Research Institute, University of California, Los Angeles, CA, United States

a r t i c l e i n f o a b s t r a c t

Article history: Recent research in instrumental conditioning has focused on the striatum, particularly the role of the dor-
Received 26 August 2008 sal striatum in the learning processes that contribute to instrumental performance in rats. This research
Received in revised form 24 October 2008 has found evidence of what appear to be parallel, functionally and anatomically distinct circuits involv-
Accepted 25 October 2008
ing dorsomedial striatum (DMS) and dorsolateral striatum (DLS) that contribute to two independent
Available online 5 November 2008
instrumental learning processes. Evidence suggests that the formation of the critical action–outcome
associations mediating goal-directed action are localized to the dorsomedial striatum, whereas the sen-
Keywords:
sorimotor connections that control the performance of habitual actions are localized to the dorsolateral
Dorsal striatum
Prefrontal cortex
striatum. In addition to the dorsal striatum, these learning processes appear to engage distinct cortico-
Instrumental conditioning striatal networks and to be embedded in a complex of converging and partially segregated loops that
Goal-directed action constitute the cortico-striatal thalamo-cortical feedback circuit. As the entry point for the basal ganglia,
Habit cortical circuits involving the dorsal striatum are clearly in a position to control a variety of motor functions
Reward but, as recent studies of various neurodegenerative disorders have made clear, they are also involved in a
Reinforcement number of cognitive and executive functions including action selection, planning, and decision-making.
© 2008 Elsevier B.V. All rights reserved.

Contents

1. Goal-directed action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
1.1. Cognition, behavioral control and Pavlovian conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
1.2. Cognitive and motivational control of actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
1.3. Neural bases of goal-directed action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
1.4. Causal learning and the cortico-striatal network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
2. Habitual action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3. The relationship of goal-directed and habit processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.1. Competition or cooperation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.2. Integration and interaction in the cortico-basal ganglia network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4. Summary and conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

In instrumental conditioning an animal’s actions are, pro- antecedents, regarding instrumental actions as a form of acquired
cedurally speaking, instrumental to the occurrence of some reflex. Thorndike [1], for example, characterized this learning as
consequence or outcome. For most of the, now quite lengthy, period ‘trial and error’ and formulated an associative theory of its acquisi-
since it was first described, however, theories of instrumental learn- tion encapsulated within the, so-called, ‘law of effect’. According
ing have referred not to the consequences of actions but to their to this view, responses in a situation that result in satisfaction
(later, more ambiguously, referred to as reinforcement, e.g. by Hull
[2]) become more firmly (and responses resulting in dissatisfac-
tion more weakly) connected with that situation; the probability
夽 The preparation of this article and the research it describes were supported by of performing a response reflects, therefore, the strength of the
grants from the NIMH #56446 and NICHD #59257. situation–response (S–R) association.
∗ Corresponding author at: Department of Psychology, UCLA, Box 951563, Los
Although there were quibbles over various aspects of S–R theory
Angeles, CA 90095-1563, United States. Tel.: +1 310 825 7560/2998;
(e.g. [3,4]), its dominance over research in instrumental condition-
fax: +1 310 206 5895.
E-mail address: balleine@psych.ucla.edu (B.W. Balleine). ing went unassailed for much of the 20th Century. Over the last

0166-4328/$ – see front matter © 2008 Elsevier B.V. All rights reserved.
doi:10.1016/j.bbr.2008.10.034
44 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
two decades, however, this theoretical framework has been sub-
stantially and quite radically revised. It is now generally accepted
that instrumental conditioning engages two distinct learning pro-
cesses, one that can be characterized in S–R terms and a second,
fundamentally different process through which animals encode
the consequences of their actions and that we have proposed is
critical to the acquisition and deployment of goal-directed actions
[5–12]. Although S–R association can sometimes dominate (when
actions become habitual), it now appears that, in most circum-
stances, the probability of a response is a product of associations
with both its antecedents and its consequences, i.e. that these learn-
ing processes can exert a cooperative influence on the selection
and initiation of instrumental actions. Indeed, at a neural level
we have argued that managing the interplay between these two
associative processes is the primary function of the basal ganglia
[11–13].
In what follows we will briefly review the behavioral and neural
evidence for these claims before considering two important issues
involving, first, evidence for the distinct sources of these influences
on performance at both a cortical and a striatal level and, second,
the evidence demonstrating their integration in implementing a
course of action and the role of the basal ganglia in this process. It
will be noted that we plan here to focus primarily on the processes
contributing specifically to instrumental conditioning. Other recent
reviews have discussed the relationship between instrumental and
Pavlovian conditioning processes and their neural bases in more
detail and the interested reader is referred there for further discus-
sion (cf. [7,14–16]).
1. Goal-directed action
1.1. Cognition, behavioral control and Pavlovian conditioning
Paradoxically, although the cognitive control of behavior has
been of increasing interest to neuroscientists, research in this area
has focused predominantly on predictive learning in Pavlovian
conditioning paradigms such as fear conditioning and eye-blink
conditioning. There is, however, no necessary relationship between
cognition and the performance of the Pavlovian conditioned
response (CR). Indeed, although it is not generally recognized,
at an adaptive level a cognitive mechanism is of little functional
value to a purely Pavlovian animal because the production of the
CR is under the control of the CS–US association and is demon-
strably not determined by a direct relationship between the CR
and the US (e.g. [17–19]). As a consequence, although the produc-
tion of the CR is clearly influenced by the nature of the CS–US
association, no amount of refinement in the cognitive represen-
tation of the CS, US or their relationship can increase the ability
of an animal to control the direction of the CR. In fact, a cognitive
mechanism can only exert a functional effect on behavior when
coupled to a process capable of modifying, withholding or reversing
the direction of actions on the basis of that information, some-
thing that demands greater behavioral control than the system
mediating Pavlovian conditioning provides (cf. [20,21] for further
discussion).
For similar reasons, the Pavlovian paradigm can provide only
a limited animal model of the effects of neuropathology on, so-
called, executive functions in humans and that evidence suggests
depend upon the integrity of various prefrontal-subcortical cir-
cuits [22–24]. Deficits in executive function have been generally
described as comprising multiple components usually including
volition, planning, and purposive action [25], capacities that fall
outside the Pavlovian domain. The limbic cortices appear to be
particularly heavily affected in executive dysfunction and several
investigators have proposed that distinct constellations of symp-
toms may reflect the disconnection of this cortical area from
specific subcortical regions such as the mediodorsal thalamus (in
Alzheimers, e.g. [26]), areas of the striatum (in Parkinsons, Hunt-
ingtons and obsessive compulsive disorders, e.g. [27–31]), and the
amygdala (in various emotional disorders, e.g. [32]). A disturbing
feature highlighted in recent work is the increasing evidence for
the early onset of many of the dysfunctions associated with these
disorders, something that suggested to Brown and Marsden [27],
amongst others, that even quite substantial motor deficits involv-
ing tremor and choreic symptoms may partially reflect a disorder
in the sustained functioning of a prefrontal-basal ganglia-cortical
feedback network engaged during planning, response selection
and initiation. However, studying normal and pathological exec-
utive functions will require models of behavioral control that
go beyond predictive learning to capture the processes engaged
during the acquisition and implementation of new behavioral
strategies.
1.2. Cognitive and motivational control of actions
Given these limits of Pavlovian processes, it is important to note
that instrumental conditioning in rodents has been found to pro-
vide an alternative and quite accurate model of executive control
generally and of human goal-directed action in particular. Models
of human action (e.g. [33–36]) have tended to focus on two critical
determinants of goal-directed actions: (1) their dependency on the
experienced causal relation between acting (or not acting) and the
occurrence of some consequence; and (2) the sensitivity of these
actions to changes in the desirability of the consequences or goal of
an action. From this perspective, actions that persist even when
causally unrelated to their consequences or when those conse-
quences are demonstrably no longer valued should not be regarded
as goal-directed.
As we pointed out some time ago [5], this “desire plus belief”
characterization of human actions can be used to distil two cri-
teria, what we have called the contingency and the goal criteria,
for the detection of goal-directed actions in any species. Since
that time we have accumulated considerable evidence suggesting
that, for the most part, the performance of instrumental actions
by rodents satisfies these criteria. Not only are these instrumental
actions highly sensitive to changes in the value of their associated
outcome, i.e. post-training devaluation often produces profound
changes in the subsequent rate of performance (cf. [6,7,10] for
reviews), there is also considerable evidence suggesting that, unlike
Pavlovian CR’s, these actions are sensitive to changes in the causal
relation to their consequences; generally, rats will stop respond-
ing if performance no longer delivers the instrumental outcome
and will stop responding even faster if their responding now can-
cels an otherwise freely available food [37,38]. Furthermore, using
a schedule developed by Hammond [39], in which the probability
of an outcome given a response (i.e. p(O/R)) and the probability
of an outcome in the absence of that response (p(O/noR)) can be
independently manipulated, we, amongst others, have reported
clear evidence that performance declines as the latter probabil-
ity increases, even when action–outcome contiguity (i.e. p(O/R)) is
kept constant and at a rate that ordinarily maintains substantial
levels of performance [8,9,40–43].
Given the clear sensitivity of actions to changes in the probabil-
ity of outcome delivery it might also be expected that performance
would also be sensitive to information concerning the likelihood
of earning a particular outcome. And, indeed, there is considerable
evidence that stimuli associated with rewarding events can exert
quite specific effects on outcome selection and on choice in studies
assessing Pavlovian-instrumental transfer. What has also become
clear, however, is that this effect does not depend on the mere asso-
 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
ciation of the CS and US but on the information that the stimulus
provides about the forthcoming US. Delamater [44], for example,
has demonstrated that, although reducing the predictive validity
of a cue with respect to the specific US with which it was associ-
ated had only a mild effect on the performance of the Pavlovian
CR, it completely abolished the influence of that cue on instrumen-
tal choice performance in a test of Pavlovian-instrumental transfer.
These data suggest that the influence of reward-related stimuli on
choice between actions is based on the information that these stim-
uli convey.
Further evidence along these lines comes from the recent find-
ing that the processes by which reward itself and stimuli that
predict reward control goal-directed action differ. Importantly,
many theories of goal-directed action either do not distinguish
between these processes or view one as ancillary to the other
[45–49]. For example, in reinforcement learning the value of the
instrumental outcome is coextensive with the value of stimuli or
states that predict that outcome and changes in performance fol-
lowing changes in value are, therefore, determined by this common
evaluative process [49–51]. This is also true of theories derived from
economics, such as utility theory, where the expected value of an
action is a product of the amount and the probability of reward or,
for expected utility, a weighted average calculated from the utility
in each state. In fact, contrary to these suggestions, the influence
of the experienced outcome value (as assessed by outcome reval-
uation) and of expected value based on cues that predict reward
(assessed using Pavlovian-instrumental transfer) on the perfor-
mance of goal-directed actions has been doubly dissociated both
neurally [52] and behaviorally [53,54].
It is interesting to note that establishing the distinct neural cir-
cuits that mediate the effects of changes in reward and of reward
prediction may provide a basis for characterizing the distinct
influences of motivational/emotional processes and of cognition
on decision-making (see particularly [55]). Whereas changes in
reward value induced by motivational manipulations are based on
changes in the emotional response associated with the rewarding
event [10,11,56], the information conveyed by cues associated with
reward, particularly with respect to the relative validity of their
predictions, may reflect the role that information regarding the con-
sequences of acting (such as that conveyed by advertising) can have
on action selection. Certainly, changes in outcome value appear to
have very little effect on the biasing effects that reward-related cues
have on choice [15,54,57]. Furthermore, consistent with the claim
that it is the information that cues provide about outcomes rather
than their ability to act as surrogates for outcome evaluation that
determine their influence on choice, several of the neural structures
found to play a role in outcome specific Pavlovian-instrumental
transfer effects, notably the basolateral amygdala, mediodorsal tha-
lamus and orbitofrontal cortex, have also been found to mediate the
predictive validity of the Pavlovian CS, i.e. lesions of these structures
each render rats insensitive to degradation of specific Pavlovian
CS–US contingencies [58,59].
1.3. Neural bases of goal-directed action
Psychologically speaking, the learning and memory processes
underlying goal-directed actions should clearly be regarded as
declarative; choice performance clearly reflects the ability of
animals to express their knowledge of various action–outcome
relations in the face of changing expectations of reward [60].
Nevertheless, despite arguments regarding the function of the
hippocampus in declarative learning of this kind [61,62,139], in
several series of studies we were unable to find any clear evi-
dence for the involvement of the hippocampus or its projections
through anterior thalamus in instrumental learning [43,63,64].
45
Table 1
Summary of the effects of excitotoxic lesions of various components of the cortico-
striatal network on sensitivity to selective contingency degradation and outcome
devaluation in instrumental conditioning.
Region Contingency degradation Outcome devaluation Reference
PL × × [9,53]
√
OFC × [15]
DMS × × [77,80]
√ √
DLS [93,101]
MDT × × [59,64]
√ √
ANT [64]
√ √
NACsh [52]
√
NACco × [52]
√ √
HPC [43,63]
√
EC × [63]
√
ACC ? [140]
√
GI × [141]
√
Abbreviations: : normal ×: deficit; PL: prelimbic area; OFC: orbitofrontal cortex;
DMS: dorsomedial striatum; DLS: dorsolateral striatum; MDT: mediodorsal tha-
lamus; ANT: anterior thalamic nuclei; NACco: nucleus accumbens core; NACsh:
nucleus accumbens shell; HPC hippocampal formation; EC: entorhinal cortex; ACC:
anterior cingulated cortex; GI: gustatory insular cortex.
These early experiments did, however, find evidence for the
involvement of the mediodorsal thalamus as well as one of its
main cortical efferents – the prelimbic region of the medial
prefrontal cortex (PL) – in this form of learning. Unlike the hip-
pocampus, cell body lesions of these areas were effective in
abolishing rats’ sensitivity to both outcome devaluation and to
selective degradation of the instrumental, action–outcome con-
tingency [8,9,53,64]. A summary of experiments assessing the
influence of pretraining lesions of various afferents and effer-
ents of the prefrontal cortex on these tests is presented in
Table 1.
Recently we have found evidence that the involvement of the
prefrontal cortex in goal-directed learning is phase-limited (see
also [65,66]). In a recent series we found clear evidence that only
damage to the PL made prior to instrumental training had any
effect on conditioning; lesions made after training was complete
had no effect on outcome devaluation [67]. This suggested to us
that, although the PL is clearly involved in goal-directed learning it
is not the locus of the action–outcome association. The PL has two
well documented striatal efferents; one arising predominantly in
layer II and projecting to the core of the nucleus accumbens [68]
and a second arising predominantly in layers V/VI and projecting to
the dorsomedial or associative striatum (DMS) [69]. The results of
other work had led us to believe that the former plays an important
role in instrumental performance but not in instrumental learning
[52]. As such we turned our attention to the other projection to the
dorsomedial striatum.
In fact, the DMS is an excellent candidate for the locus of the
plasticity mediating the encoding of the action–outcome associ-
ation in instrumental conditioning. As illustrated in Fig. 1, it is a
critical component in the associative cortico-basal ganglia circuit
and receives inputs from association cortices such as the PL as
well as the premotor or medial agranular cortex involved in the
action monitoring and programming implicated in executive pro-
cesses [70,71] and projections from the DMS are in a position to
influence downstream motor control networks in the brainstem
as well as the motor thalamo-cortical reentrant network [70]. The
posterior part of the DMS also receives inputs from the basolat-
eral amygdala [72], a structure that, according to recent evidence,
mediates the assignment of incentive value to the consequences of
instrumental actions [73,74]. In accord with this suggestion, elec-
trophysiological studies measuring neural activity in the associative
striatum or caudate nucleus in primates, the homologue of the DMS
46 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
Fig. 1. Cortico-striatal circuits involved in instrumental conditioning. (1) The learn-
ing processes controlling the acquisition of reward-related actions are mediated
by converging projections from regions of prelimbic prefrontal cortex (PL) to the
rodent dorsomedial striatum or primate dorsoanterior striatum (DM), whereas
(2) the processes mediating the acquisition of stimulus-bound actions, or habits,
are thought to be mediated by projections from sensorimotor cortex (SM) to the
rodent dorsolateral–primate dorsoposterior striatum (DL). These cortico-striatal
connections form parts of distinct feedback loops that project back to their cor-
tical origins via the substantial nigra/globus pallidus (SNr/Gpi) and the mediodorsal
(MD)/posterior (PO) nuclei of the thalamus. (3) Reward and predictors of reward are
the major motivational influences on the performance of goal-directed and habit-
ual actions that are thought to be mediated by cortico-striatal circuits involving,
particularly, ventral striatum (VS) and regions of the amygdala; the basolateral area
(BLA) for goal-directed actions and the central nucleus (CeN), via its modulation of
the dopamine neurons in the substantia nigra pars compacta (SNc). Dopamine is an
important modulator of plasticity in the dorsal striatum whereas its tonic release
has long been associated with the motivational processes mediated by the ventral
circuit.
in rats, have reported that neural activity in this region correlated
with the performance of motor movements and can be modulated
by the expectancy of reward [75,76]. More directly, in a recent
series of experiments we found direct evidence that, in contrast
to manipulations of prefrontal cortex, both pre- and post-training
cell body lesions of the DMS as well as local inactivation of this area
induced by infusions of the GABA-A agonist muscimol, reduced the
sensitivity of rats’ instrumental performance both to shifts in the
action–outcome contingency and to post-training outcome deval-
uation [77].
The suggestion that the DMS is the locus of action–outcome
encoding in instrumental conditioning contrasts with other recent
claims that the ventral [78] or the posterolateral striatum [79]
mediates learning critical to the acquisition of goal-directed
actions. Nevertheless, these studies only assessed changes in
instrumental performance and did not directly assess changes in
the content of learning. Furthermore, as presented in Table 1, we
have found that lesions of the ventral striatum, although some-
times effective in influencing performance, do not affect the rat’s
sensitivity to changes in the action–outcome contingency. In a sec-
ond recent series, however, we used well-established behavioral
assays that unambiguously distinguish action–outcome learning
from other types of learning to assess the role of the DMS in the
formation of action–outcome associations [80]. Given the evidence
that NMDA receptor (NMDAR) activation is involved in long-term
plasticity such as long-term potentiation in the dorsal striatum
[81,135], we proposed that action–outcome encoding requires acti-
vation of NMDARs in the DMS. This hypothesis was tested in rats
that, after a period of pre-training, were given a bilateral infusion of
either a selective NMDAR antagonist (APV), or vehicle prior to a sin-
gle learning session in which they were trained to press two levers
for distinct food outcomes. The next day the rats were tested using
an outcome devaluation protocol, i.e. they were allowed to consume
one of the two outcomes for 1 hr before a choice extinction test was
given on the two levers. We found, first that APV immediately prior
to training did not affect performance either during training or test-
ing but strongly attenuated the ability of the rats to use changes in
outcome value to modify their instrumental performance, i.e. they
appeared not to have encoded the specific action–outcome associ-
ations to which they were exposed during training. Furthermore,
in subsequent experiments we found both that APV infused imme-
diately after training did not have this effect on action–outcome
encoding, nor did the infusion of APV into adjacent dorsolateral
striatum (DLS) [80].
1.4. Causal learning and the cortico-striatal network
One element of the rodent data that has remained open to
critique has been the difficulty convincingly to demonstrate the
active involvement of this prefrontal cortical–dorsomedial stri-
atal network in the translation of causal knowledge about actions
and their consequences into goal-directed learning. In a recent
study, however, we tried to address this directly not in rats but in
human subjects [82]. We trained humans on a free-operant button-
pressing task in which they could earn money as the outcome and
from these sessions we extracted both an objective measure of the
instrumental contingency (i.e. the rate of button pressing and of
outcome delivery through time) and a subjective measure that we
took by asking the subjects to rate on a 100 point scale how causal
they thought their actions were. We gave the subjects a number of
sessions of training and compared the variation in these measures
of the objective and subjective contingency as well as changes in
the activity of neural structures by scanning subjects using func-
tional magnetic resonance imaging while they pressed a button
and earned money as this response–reward relationship changed
over time.
We found that the subjects’ judgments about the causal effi-
cacy of their actions varied positively and significantly with the
objective contingency between the rate of button pressing and the
amount of money they earned. Furthermore, as we reported in the
rodent, neural responses in medial frontal cortex and dorsomedial
striatum were modulated as a function of contingency and these
regions altered their activity during periods when actions were
highly causal compared with when they were not. Moreover, we
found that the medial prefrontal cortex tracked local changes in
the correlation of action and outcome rates, implicating this region
in the on-line computation of contingency [82]. Hence, just as we
have found in rodents, in this study a network involving the medial
prefrontal and medial orbital cortices together with the dorsome-
dial striatum was implicated in detecting the causal relationship
between actions and their consequences consistent with the claim
 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
that these structures act together to during the acquisition of goal-
directed actions to encode specific action–outcome associations.
2. Habitual action
Input to the striatum from sensorimotor cortex, particularly pri-
mary motor and somatosensory cortices (cf. [83,84]), appears to be
involved in what is commonly thought to be a completely differ-
ent form of performance process involving the control of actions by
antecedent stimuli through a traditional S–R/reinforcement asso-
ciative mechanism. In recent years this process has been argued
to form the core of a distinct functional capacity involving the for-
mation of habitual actions. Habits are revealed particularly in their
persistence, even in the face of sometimes quite extreme negative
consequences, and in their sensitivity to the motivational func-
tions of reward-related cues [85]. Many of the ideas that have been
expressed in recent papers, particularly those linking habit learning
with various addictive behaviors (e.g. [86–89,142]), have their root
in now classical theories of habit learning, associated most notably
with Hull [2], that explain the acquisition of actions instrumental
to gaining access to rewarding events in terms of the operation of
a S–R/reinforcement architecture.
From this perspective, rewarding events reinforce or strengthen
associations between contiguously active sensory and motor pro-
cesses allowing the sensory process to elicit the motor response
in a manner that is no longer regulated by its consequences. As
such, the determining features of habit learning can be readily
distinguished from those of goal-directed action: (1) the lack of
regulation by the consequences of an action suggests that habitual
actions should be insensitive to post-training outcome devaluation;
(2) the emphasis on stimulus-response contiguity suggests that,
rather than reflecting the operation of an error correction learn-
ing rule, a particular S–R association is strengthened whenever the
response is reinforced in the stimulus, irrespective of the specific
outcome delivered or other stimuli present.
In line with these proposed features of habits, there is consid-
erable evidence that when either overtrained [90] or trained on
interval schedules of reinforcement [91,92] – i.e. whenever changes
in the rate of reward are constrained – instrumental performance
becomes insensitive to changes in outcome value. Likewise, evi-
dence suggests that, unlike goal-directed actions, habitual actions
are relatively insensitive to changes in the action–outcome contin-
gency. For example, Dickinson et al. [38] found that, when asked
to withhold the performance of a previously reinforced lever press
action in order to get access to sugar, rats that had been under-
trained were able to do so whereas those that had been overtrained
were not. This finding has been replicated both in a different kind
of training situation [93] and in a different species (i.e. mice; [94])
and suggests, in line with the features described above, that habit-
ual actions are relatively insensitive to changes in the relationship
between action and outcome.
Interestingly, evidence suggests that a cortico-striatal network
parallel to that implicated in goal-directed action involving senso-
rimotor cortices together with the dorsolateral striatum in rodents
may mediate the transition to habitual decision processes asso-
ciated with S–R learning (see Fig. 1; [95,96]). Changes in the
DLS appear to be training related [97–99] and to be coupled to
changes in plasticity as behavioral processes become less flexible
[99,100]. Correspondingly, whereas overtraining causes perfor-
mance to become insensitive to outcome devaluation, we have
found that lesions of DLS reverse this effect rendering performance
once again sensitive to devaluation treatments [101]. Likewise, we
have found that muscimol inactivation of DLS renders otherwise
habitual performance sensitive to changes in the action–outcome
contingency [80].
47
It is worth noting, however, that, because these are essentially
tests for features of habits that distinguish them from goal-directed
actions, they establish evidence for habits by default and, while the
strength of the S–R association likely correlates strongly with the
degree of insensitivity to devaluation and contingency manipula-
tions, insensitivity to these manipulations does not directly assess
S–R learning per se. However, positive evidence for the involvement
of the DLS in S–R learning has been reported using instrumental
conditioning procedures that encourage animals to form S–R rela-
tionships in order to solve complex discrimination problems. For
example, Featherstone and McDonald [102,103] have shown that
lesions of the DLS impair both the acquisition and performance
of a simple discrimination task in which lever presses are rein-
forced during presentations of one stimulus (S+) but not during
presentations of another stimulus (S−). Consistent with both an
S–R structure and with the putative role of the DLS in this associa-
tive process, a salient feature of stimulus-controlled instrumental
performance is the general failure to respond that can be generated
by DLS lesions (see, for example, [103]) in contrast to their effects
on simple free-operant tasks (cf. [101]). Indeed, studies assessing
the role of the striatum in basic motor behaviors have shown that
lateral striatal lesions can cause severe impairments in both the ini-
tiation and amplitude of movements such as forelimb reaches (e.g.
[138]). However, these impairments of basic movement can make
the results of experiments aimed at investigating specific learn-
ing deficits difficult to interpret. For example, in Featherstone and
McDonald [103] assessment of the influence of post-training DLS
lesions on a simple discrimination, DLS lesioned animals responded
significantly less than sham controls on S+ trials, and did not dif-
fer from the sham group in their responding on S− trials. Although
consistent with a deficit in stimulus-controlled actions, it is also
possible, because post-training responding during the S− was close
to zero in both groups, that the lack of a difference during the S−
reflected a floor-effect.
In an attempt better to distinguish failures to respond from
failures of stimulus control over actions during this kind of discrim-
ination task, we trained animals on a conditional discrimination
in which the two discriminative stimuli supported equal lev-
els of responding but on different levers. We then assessed the
dose-dependent influence of muscimol-induced inactivation of the
DLS on response initiation and discriminative accuracy (correct
responses/total responses). During training, rats were required to
press the right lever (R1) in response to one auditory cue (S1) and
the left lever (R2) in response to a second auditory cue (S2). All
correct lever presses were reinforced with a grain pellet outcome
(O1), and trials were terminated (i.e. levers retracted) by the first
response, whether it was correct or incorrect, and training contin-
ued until all animals had reached at least 70% accuracy. As such
the structure of the task constituted a bidirectional discrimination
problem, viz:
S1 : R1–O1, R2–; S2 : R1–, R2–O1
Although it is possible that animals form hierarchical S:R–O
associations to solve this task, the simplest solution involves form-
ing two simple S–R associations: S1–R1 and S2–R2. Of course,
to the extent that rats utilize conditional R–O associations we
should not anticipate effects of DLS inactivation on this task. If,
however, the rats acquired and utilized the simpler S–R solution
to this problem then we should find evidence specifically of a
failure of discrimination after DLS inactivation resulting from an
inactivation-induced inability to use the S–R associations encoded
during training.
To assess this prediction we conducted three separate tests in
each of which the Long–Evans rats that we used as subjects (n = 10)
48 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
Fig. 2. Effect of inactivation of the dorsolateral striatum using muscimol at high
(0.5 ␮g) and low (0.25 ␮g) doses and of vehicle infusion on discrimination accuracy
(top panel) and on response initiation (bottom panel) in a biconditional discrimina-
tion task – see text for details.
were required to perform the discrimination task in each of three
conditions: after bilateral infusion into the DLS of a high dose of
muscimol (0.5 ␮g, 0.5 ␮l per hemisphere), a low dose of muscimol
(0.25 ␮g, 0.5 ␮l per hemisphere) or of vehicle (0.5 ␮l per hemi-
sphere), in counterbalanced order. Fig. 2 shows the discrimination
accuracy and initiation failures in the upper and lower panels,
respectively, for the three test conditions. ANOVA found an effect
of drug dose on both discrimination, F(2, 18) = 23.0, and response
initiation, F(2, 18) = 27.7. Nevertheless, as is clear from Fig. 2, these
effects altered across the course of testing. In the high dose con-
dition a clear and immediate loss of discrimination accuracy was
observed. Although this was difficult to dissociate from the per-
formance effects of the muscimol infusion on response initiation,
it should be noted that, for the first 20 trials or so, the rats were
responding on more than half the trials and yet showed no evi-
dence of accurate discrimination. This pattern was even clearer at
the lower dose where we found that discrimination accuracy was
reduced significantly by the second block of trials (p < 0.05) at a
point when response initiation did not differ from vehicle controls
(p > 0.05). From this point discrimination at the low dose fell essen-
tially to chance levels with only very minimal effects on response
initiation; by the third or fourth block of 10 trials during the test,
the rats were completing approximately 75% of the trials and yet
their discrimination accuracy had fallen to chance and did not differ
from accuracy in the high dose group (p > 0.05). Indeed, both groups
differed from the vehicle controls from the second block onwards
(all p’s < 0.05).
These results allow us to more conclusively separate discrimi-
nation failures from more general initiation failures, and provide
support for the predicted role of the DLS in performance on
this bidirectional discrimination. Furthermore, these data sug-
gest, at least at the lower dose level, that the DLS mediates
the accuracy of the discrimination over and above the changes
in performance induced by DLS inactivation. Hence, at a point
at which the rats were showing relatively low levels of motor
impairment they were unable to use the discriminative stimuli
to select the reinforced response. These results are consistent
with the rats using an S–R strategy to solve this complex dis-
crimination and with the hypothesized involvement of the DLS
in the encoding of S–R associations and their expression in
performance.
3. The relationship of goal-directed and habit processes
3.1. Competition or cooperation?
Together, the findings from the experimental investigations of
the dorsal striatum described above have identified two distinct
functional systems within adjacent regions of dorsal striatum:
specifically, a circuit mediating goal-directed learning and involv-
ing the dorsomedial striatum and a circuit mediating habit or
procedural learning and involving the dorsolateral striatum. Fur-
thermore, at least at the level of the striatum these functions
appear to be independent; damage to dorsolateral but not dorso-
medial striatum renders otherwise habitual actions goal-directed
whereas damage to the dorsomedial striatum renders otherwise
goal-directed actions habitual. It appears, therefore, that these two
regions of the striatum, or, perhaps more accurately, the distinct
cortico-striatal circuits involving these regions, may compete for
control of instrumental performance.
There are however, layers of complexity in attempting to under-
stand the interaction of these apparently distinct action controllers.
At one level it is clear that, at least in habitual actions, the
R–O goal-directed process and the S–R habit process compete for
control of performance; habitual control can apparently be imme-
diately released and an underlying goal-directed control revealed
by muscimol infusion into the DLS [93]. In fact even under normal
circumstances evidence suggests that the goal-directed process can
quickly suppress habitual control. This can be noticed in the every-
day, e.g. while driving on a freeway when, after a period of carefree,
apparently cognitively disconnected driving we see a police car
approaching in the rearview mirror. Do we carry on driving in so
carefree a manner? Not likely; even if we are within the speed
limit and generally obeying the rules of the road, our vigilance
is increased and our driving becomes more deliberated; the habit
has been suppressed. Likewise, rats that are behaving habitually
in extinction and so responding at a high rate on a lever trained
with a sucrose solution that has subsequently been devalued, will
stop responding as rapidly as non-habitual lever pressers when the
lever response is punished by the actual delivery of the devalued
outcome (e.g. [92,104]). The rapidity of this adjustment is, how-
ever, severely curtailed by damage or inactivation of dorsomedial
striatum [77], a finding that is consistent with the argument that
it is the return of control by the goal-directed system that is the
source of rapid suppression of the habit in the punishment situa-
tion.
 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
Some forms of psychopathology, most notably drug addiction,
might well find their source in a defective ability to suppress
habitual actions by re-engaging the goal-directed system. Dur-
ing the development of addiction, the pursuit of drugs of abuse
rapidly becomes habitual coming under the control of internal and
external states and stimuli rather than the consequences of act-
ing [86,105,106]. It is important, however, to distinguish habitual
drug seeking from other forms of habitual behavior. Under nor-
mal conditions, habit learning can be highly adaptive; habits allow
us and other animals to relegate the control of routine behavioral
responses to a system that uses few cognitive resources, freeing
up a limited executive capacity for tasks that need greater mon-
itoring. In contrast, habitual drug seeking is pathological; drug
exposure increases the rate of acquisition of habitual actions and
the influence of drug associated contexts and cues on their per-
formance. Furthermore, despite the heavy emphasis on habit in
current research on drug addiction, a distinguishing feature of
habitual drug seeking is the addicts’ loss of executive control over
the habit. As is commonly noted (e.g. [107]), a distinguishing fea-
ture of drug seeking is its persistence in the face of severe negative
consequences (cf. DSM IV criteria for drug abuse). The compul-
sive pursuit of drugs can be viewed, therefore, as the product
of a drug-induced increment in habit acquisition and a drug-
induced decrement in the addict’s ability to exert control over the
habit in the face of persistent, negative feedback. Indeed, consis-
tent with this argument, Robinson and co-workers have reported
structural changes involving a loss of dendritic spines induced
by sensitization to methamphetamine in dorsomedial striatum, a
key structure implicated in goal-directed action in rodents, and
a concomitant increase in spine density in dorsolateral striatum
[108].1
On the other hand, there is evidence that S–R and R–O learn-
ing processes cooperate in the integration of stimulus-mediated
action selection with action evaluation processes during the initia-
tion of goal-directed actions [12,58]. Perhaps the strongest evidence
for cooperation of this kind comes from studies of outcome selec-
tive reinstatement in which we have found that, when previously
trained on two actions for distinct outcomes, the delivery of one
or other outcome after a period of extinction on the two actions
results in the reinstatement of the action that, in training followed
the delivery of that outcome rather than the reinstatement of the
action that delivered that outcome [58]. As we have described in
more detail previously [12], the ability of outcomes to exert this
effect on response selection is not affected by devaluation of the
reinstating outcome. Nevertheless, using a similar training situa-
tion in which outcomes were used as explicit discriminative cues
for action selection, we found that these kind of stimuli can, in
fact, engage an evaluative process based on the R–O association,
rendering the rate or vigor of performance of an action sensitive
to the current value of the outcome that the action earned during
training, i.e. the rate of performance, but not the action selection, is
attenuated if the outcome earned by the reinstated action is deval-
ued ([143]; see [58] for related findings). Hence, it appears that, in
the ordinary course of events, (1) a form of stimulus-response pro-
1
It is worth noting here – as an aside – that this account overcomes some of
the general problems identified with the purely habit based account of drug addic-
tion. For example, one argument against the claim that drug addiction reflects an
abnormal increment in habit learning has been based on, albeit largely anecdo-
tal, evidence of the highly devious and nefarious strategies that addicts devise in
procuring drugs. The alternative perspective proposed here sidesteps this kind of
issue by emphasizing the pathological nature of the habitual control induced, not
simply by an increment in habit learning but by drug-induced abnormalities in the
goal-directed system with the consequent changes in goal-directed decision-making
processes and in behavioral control.
49
cess lies at the heart of action selection, that (2) action selection
causes the retrieval of the outcome associated with that action,
based on the action–outcome association, and, consequently, (3)
the retrieval of the incentive value of that outcome and that these
later action–outcome and outcome value processes are a neces-
sary step towards the actual performance of the action. Hence,
this selection–evaluation–initiation sequence appears to require the
cooperative integration of the S–R and R–O learning processes.
From this perspective, therefore, choice and decision-making is an
integrative process.
It is, of course, possible that both competition and cooperation
between goal-directed and habit learning processes occurs but at
different times or under different conditions. For example, it is
possible that R–O and S–R processes ordinarily cooperate but that
overtraining provides the conditions under which stimuli not only
exert control over action selection but also dominate action initia-
tion; the strength of the stimulus-response association may allow
the action to be performed before it is properly evaluated, some-
thing that accords well with the notion that habitual actions are
relatively impulsive. Likewise, whilst otherwise cooperative, the
inhibition of habits may be a function of the quite specific condi-
tions induced, for example, by the delivery of unexpected negative
feedback. Nevertheless, the fact that both the goal-directed and
habit learning systems appear to be able to function without the
other suggests that their cooperation is not necessary for instru-
mental performance, although it may be necessary for actions to
adjust normally to changes in contingency and so for instrumental
performance to remain adaptive when conditions are particularly
volatile.
3.2. Integration and interaction in the cortico-basal ganglia
network
It is not clear exactly how competition and cooperation is
realized in the neural networks mediating action–outcome and
stimulus-response learning. Certainly, the anatomy of the cortico-
basal ganglia network provides virtually limitless possibilities for
convergence, divergence, integration and interaction between the
complex functions that appear to be instantiated in this circuitry
and, indeed, there have been many different hypotheses advanced
based on this anatomy. For example, segregation of function falls
naturally from the description of parallel feedback loops connect-
ing discrete regions of the cortex with striatum, midbrain, thalamus
and feeding back to their cortical origin [109,110]. Indeed, this kind
of account is well suited to the suggestion that cognitive and exec-
utive functions, including goal-directed action, are mediated by the
active maintenance of patterns of neural activity in different regions
of prefrontal cortex [45,111–117]. Nevertheless, evidence that the
prefrontal cortex plays only a time-limited role in goal-directed
learning is not predicted on this account; the loop appears to be
curtailed after the new strategy has been encoded.
It is, however, possible to propose alternative hypotheses
to account for these data; for example, it is possible that the
action–outcome association is encoded in the medial prefrontal
cortex and then consolidated in the dorsomedial striatum through
direct connections between the PL and the DMS that consti-
tute the medial loop [98].2 Alternatively, it is worth noting that
the description of this loop-like architecture superseded the ear-
lier quite attractive idea of functional integration in the striatum
through the convergence of diffuse cortical regions onto a discrete
2
Although this hypothesis appears contrary to the findings of [66], note that it is
focused on the acquisition of new goal-directed actions and not on the reorganiza-
tion of cortex induced by 9 months of training on discrimination reversals.
50 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
striatal targets [118]. And, indeed, despite the shift in emphasis
onto cortical encoding processes, there remains strong evidence
for the convergence between neurons in disparate cortical areas
onto single medium spiny neurons in the dorsal striatum [119].
Unfortunately, the failure to find evidence that different regions
of cortex mediate a common function stands against this account.
With regard to rodent instrumental conditioning, for example,
evidence suggests that whereas, as described above, prelimbic
prefrontal cortex is involved in encoding action–outcome associ-
ations, the orbitofrontal cortex is not but rather plays a role in
the cognitive control of action selection on the basis of reward-
related cues [15,120], the anterior cingulate cortex plays a role in
the resolution of conflicts in action selection [143], medial agran-
ular cortex in encoding action sequences [121] and the extensive
motor and sensorimotor regions of frontal cortex appear to be pri-
marily involved in S–R learning and, hence, in stimulus-mediated
action selection. Although these findings are not consistent with the
convergence theory of the cortico-striatal network, the apparent
independent functions subserved by regions of prefrontal cortex
in instrumental conditioning is consistent with a parallel organi-
zation of cortico-striatal circuits. The twin notions of divergence
and convergence could, however, be taken to suggest that there are
some functions maintained in parallel networks whereas others are
mediated by converging inputs to striatum or, alternatively, that
some basic independent functions are encoded in distinct striato-
nigro-thalamic networks and are integrated through convergence
to allow anatomically distinct parallel circuits to generate larger
functional units [122].
Over and above cortico-striatal convergence, there has been
considerable recent interest in the role of the midbrain dopamine
projection to the striatum in the control of distinct forms of plas-
ticity and in the transition between different forms of behavioral
control. Haber et al. [123] description of a spiraling feedback
network involving ventral striatum, ventral tegmentum, dorsal
striatum and substantia nigra has, for example, been argued to
provide the basis for transitions between goal-directed and habit-
ual processes, and some recent evidence suggests that there may
be some functional relationship in this network, at least between
the ventral striatum and the dorsolateral striatum [123,136]. But, of
course, there are many other possible routes through which these
structures might interact including projections into dorsal striatum
from the ventral striato-pallido-thalamic pathway [124], through
the thalamo-striatal pathway generally [125], not to mention the
opportunity for integration and interaction in output regions such
as the globus pallidus where colaterals from both the dorsomedial
and lateral regions have been found to converge [126,127]. These
aspects of the broader basal ganglia-thalamo-cortical network have
not been systematically assessed functionally and constitute com-
plex but important targets for future studies.
4. Summary and conclusion
Whatever the neural bases of the interaction between goal-
directed and habitual processes turns out to be, recent data suggest
that the basal ganglia are able to maintain these functions in par-
allel and allow, under some conditions, one or other process either
independent control or, under other conditions, both processes
to exert cooperative control over the performance of instrumen-
tal actions. It is important to note that, in suggesting that two
distinct learning processes are concurrently engaged, this view
implies that the representation of the instrumental outcome plays
two distinct functions serving both as a reward or goal, as a part
of the action–outcome association underlying goal-directed learn-
ing, and also to reinforce an association between the action and
antecedent stimuli in habits. How this is achieved is not fully
understood at present, and space considerations preclude a full
consideration of this issue here (cf. [11,16] for reviews), but at
present it appears likely that the function of parsing the outcome
into both a reward and a reinforcement signal depends on the amyg-
dala.
In recent years considerable evidence has accumulated sug-
gesting that the basolateral amygdala plays a central role in
encoding the incentive or reward value of the instrumental out-
come and, hence, in controlling the performance of goal-directed
actions based on the interaction of this evaluative process with
the action–outcome association [59,73,74]. Likewise, a number of
authors have suggested that the reinforcement signal mediating
the acquisition of S–R associations involving the dorsolateral stria-
tum involves the ascending dopaminergic projection arising in the
substantia nigra [128,129], a projection that appears to be at least
partly controlled by the central nucleus of the amygdala [130].
Although direct evidence that the CeN plays a role in the rein-
forcement signal has not yet been reported, it is known to be
involved in generating general affective responses to rewarding
events [144], signals associated with rewarding events [131,132]
and in the control of simple stimulus-response associations, such
as those involving the performance of orienting responses to
stimuli associated with food [133,134,137]. By activating both the
central and basolateral amygdala, therefore, a single outcome-
related event could potentially exert distinct functional effects in
instrumental conditioning by controlling the production of inde-
pendent reward and reinforcement signals that concatenate to
distinct regions of striatum to control distinct cortico-striatal cir-
cuits.
References
[1] Thorndike EL. Animal intelligence: experimental studies. New York: Macmil-
lan; 1911.
[2] Hull CL. Principles of behavior. New York: Appleton; 1943.
[3] Guthrie ER. The psychology of learning. New York: Harpers; 1935.
[4] Spence KW. Behavior theory and conditioning. New Haven: Yale University
Press; 1956.
[5] Dickinson A, Balleine BW. Actions and responses: the dual psychology of
behaviour. In: Eilan N, McCarthy R, Brewer MW, editors. Spatial represen-
tation. Oxford: Basil Blackwell Ltd.; 1993. p. 277–93.
[6] Dickinson A, Balleine BW. Motivational control of goal-directed action. Anim
Learn Behav 1994;22:1–18.
[7] Dickinson A, Balleine BW. The role of learning in motivation. In: Gallistel
CR, editor. Learning, motivation & emotion, vol. 3: Steven’s handbook of
experimental psychology. 3rd ed. New York: John Wiley & Sons; 2002. p.
497–533.
[8] Balleine BW, Dickinson A. The role of incentive learning in instrumental out-
come revaluation by specific satiety. Anim Learn Behav 1998;26:46–59.
[9] Balleine BW, Dickinson A. Goal-directed instrumental action: contingency
and incentive learning and their cortical substrates. Neuropharmacology
1998;37:407–19.
[10] Balleine BW. Incentive processes in instrumental conditioning. In: Mowrer R,
Klein S, editors. Handbook of contemporary learning theories. Hillsdale, NJ:
LEA; 2001. p. 307–66.
[11] Balleine BW. Neural bases of food-seeking: affect, arousal and reward in cor-
ticostriatolimbic circuits. Physiol Behav 2005;86:717–30.
[12] Balleine BW, Ostlund SB. Still at the choicepoint: action selection and initiation
in instrumental conditioning. Ann N Y Acad Sci 2007;1104:147–71.
[13] Balleine BW, Daw ND, O’Doherty J. Multiple forms of value learning and the
function of dopamine. In: Glimcher P, Camerer C, Fehr E, Poldrack R, editors.
Neuroeconomics: decision making and the brain. Academic Press; 2008.
[14] Yin HH, Ostlund SB, Balleine BW. Reward-guided learning beyond dopamine
in the nucleus accumbens: the integrative functions of cortico-basal ganglia
networks. Eur J Neurosci 2008;28:1437–48.
[15] Ostlund SB, Balleine BW. Orbitofrontal cortex mediates outcome encoding in
Pavlovian but not instrumental learning. J Neurosci 2007;27:4819–25.
[16] Balleine BW, Killcross AS. Parallel incentive processing: an integrated view of
amygdala function. Trends Neurosci 2006;29:272–9.
[17] Konorski J, Miller S. On two types of conditioned reflex. J Gen Psychol
1937;16:264–72.
[18] Sheffield FD. Relation between classical and instrumental conditioning. In:
Prokasy WF, editor. Classical conditioning. New York, NY: Appelton-Century-
Crofts; 1965. p. 302–22.
 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
[19] Holland PC. Differential effects of omission contingencies on various compo-
nents of Pavlovian appetitive conditioned responding in rats. J Exp Psychol
Anim Behav Process 1979;5:178–93.
[20] Hershberger WA. An approach through the looking-glass. Anim Learn Behav
1986;14:443–51.
[21] Dickinson A. Expectancy theory in animal conditioning. In: Klein SB, Mowrer
RR, editors. Contemporary learning theories: Pavlovian conditioning and the
status of traditional learning theories. Hillsdale, NJ: Lawrence Erlbaum Asso-
ciates; 1989. p. 279–308.
[22] Frith CD, Friston K, Liddle PF, Frackowiak RS. Willed action and the prefrontal
cortex in man: a study with PET. Proc R Soc Lond 1991;244:241–6.
[23] Knight RT, Grabowecky MF, Scabini D. Role of human prefrontal cortex in
attention control. Adv Neurol 1995;66:21–34.
[24] Owen AM. Cognitive planning in humans: neuropsychological, neuroanatom-
ical and neuropharmacological perspectives. Prog Neurobiol 1997;53:431–
50.
[25] Lezak MD. Neuropsychological assessment. 3rd ed. New York: Oxford Univer-
sity Press; 1995.
[26] Chu C-C, Tranel D, Damasio AR, Van Hoesen GW. The autonomic-related cor-
tex: pathology in Alzheimer’s disease. Cereb Cortex 1997;7:86–95.
[27] Brown P, Marsden CD. What do the basal ganglia do? Lancet 1998;351:1801–4.
[28] Rapoport JL, Fiske A. The new biology of obsessive-compulsive disorder:
implications for evolutionary psychology. Perspect Biol Med 1998;41:159–
75.
[29] Robinson D, Wu H, Munne RA, Ashtari M, Alvir JM, Lerner G, et al. Reduced cau-
date nucleus volume in obsessive-compulsive disorder. Arch Gen Psychiatry
1995;52:393–8.
[30] Bloch MH, Leckman JF, Zhu H, Peterson BS. Caudate volumes in childhood
predict symptom severity in adults with Tourette syndrome. Neurology
2005;65:1253–8.
[31] Hodges A, Strand AD, Aragaki AK, Kuhn A, Sengstag T, Hughes G, et al. Regional
and cellular gene expression changes in human Huntington’s disease brain.
Hum Mol Genet 2006;15:965–77.
[32] Damasio AR. The somatic marker hypothesis and the possible functions of the
prefrontal cortex. Philos Trans R Soc Lond B Biol Sci 1996;351:1413–20.
[33] Dretske F. Explaining behavior: reasons in a world of causes. Cambridge, MA:
The MIT Press; 1988.
[34] Bratman ME. Intention, plans and practical reason. Cambridge: Harvard Uni-
versity Press; 1987.
[35] Frese M, Sabini J. Goal directed behavior: the concept of action in psychology.
Hillsdale, NJ: Lawrence Erlbaum Associates; 1985.
[36] Woodfield A. Teleology. New York: Cambridge University Press; 1976.
[37] Davis J, Bitterman ME. Differential reinforcement of other behavior (DRO): a
yoked-control comparison. J Exp Anal Behav 1971;15:237–41.
[38] Dickinson A, Squire S, Varga Z, Smith JW. Omission learning after instrumental
pretraining. Q J Exp Psychol 1998;51B:271–86.
[39] Hammond LJ. The effects of contingencies upon appetitive conditioning of
free-operant behavior. J Exp Anal Behav 1980;34:297–304.
[40] Colwill RM, Rescorla RA. Associative structures in instrumental learning. In:
Bower GH, editor. The psychology of learning and motivation, 20. Orlando, FL:
Academic Press; 1986. p. 55–104.
[41] Dickinson A, Mulatero CW. Reinforcer specificity of the suppression of
instrumental performance on a non-contingent schedule. Behav Process
1989;19:167–80.
[42] Williams BA. The effect of response contingency and reinforcement iden-
tity on response suppression by alternative reinforcement. Learn Motiv
1989;20:204–24.
[43] Corbit LH, Balleine BW. The role of the hippocampus in instrumental condi-
tioning. J Neurosci 2000;20:4233–9.
[44] Delamater AR. Outcome-selective effects of intertrial reinforcement in Pavlo-
vian appetitive conditioning with rats. Anim Learn Behav 1995;23:31–9.
[45] Miller EK, Cohen JD. An integrative theory of prefrontal cortex function. Annu
Rev Neurosci 2001;24:167–202.
[46] Cohen JD, Braver TS, Brown JW. Computational perspectives on dopamine
function in prefrontal cortex. Curr Opin Neurobiol 2002;12:223–9.
[47] Aston-Jones G, Cohen JD. Adaptive gain and the role of the locus
coeruleus-norepinephrine system in optimal performance. J Comp Neurol
2005;493:99–110.
[48] Schultz W. Behavioral theories and the neurophysiology of reward. Annu Rev
Psychol 2006;57:87–115.
[49] Daw ND, Niv Y, Dayan P. Uncertainty-based competition between pre-
frontal and dorsolateral striatal systems for behavioral control. Nat Neurosci
2005;8:1704–11.
[50] Daw ND, Doya K. The computational neurobiology of learning and reward.
Curr Opin Neurobiol 2006;16:199–204.
[51] Niv Y, Joel D, Dayan P. A normative perspective on motivation. Trends Cogn
Sci 2006;10:375–81.
[52] Corbit L, Muir J, Balleine BW. The role of the nucleus accumbens in instrumen-
tal conditioning: evidence for a functional dissociation between accumbens
core and shell. J Neurosci 2001;21:3251–60.
[53] Corbit LH, Balleine BW. Pavlovian and instrumental incentive processes have
dissociable effects on components of a heterogeneous instrumental chain. J
Exp Psychol Anim Behav Process 2003;29:99–106.
[54] Holland PC. Relations between Pavlovian-instrumental transfer and reinforcer
devaluation. J Exp Psychol Anim Behav Process 2004;30:104–17.
51
[55] Seeley WW, Menon V, Schatzberg AF, Keller K, Glover GH, Kenna H, et al. Dis-
sociable intrinsic connectivity networks for salience processing and executive
control. J Neurosci 2007;27:2349–56.
[56] Balleine BW. Incentive behavior. In: Whishaw IQ, Kolb B, editors. The behavior
of the laboratory rat: a handbook with tests. Oxford: Oxford University Press;
2004. p. 436–46.
[57] Rescorla RA. Transfer of instrumental control mediated by a devalued out-
come. Anim Learn Behav 1994;22:27–33.
[58] Ostlund SB, Balleine BW. Selective reinstatement of instrumental perfor-
mance depends on the discriminative stimulus properties of the mediating
outcome. Learn Behav 2007;35:43–52.
[59] Ostlund SB, Balleine BW. Differential involvement of the basolateral amyg-
dala and mediodorsal thalamus in instrumental action selection. J Neurosci
2008;28:4398–405.
[60] Winograd T. Frame representations and the declarative-procedural contro-
versy. In: Bobrow D, Collins A, editors. Representation and understanding:
studies in cognitive science. San Diego, CA: Academic Press; 1975. p. 185–
210.
[61] Squire LR. Memory and the hippocampus: a synthesis from findings with rats,
monkeys and humans. Psychol Rev 1992;99:195–231.
[62] Eichenbaum H, Schoenbaum G, Young B, Bunsey M. Functional organization
of the hippocampal memory system. Proc Natl Acad Sci 1996;93:13500–7.
[63] Corbit LH, Ostlund SB, Balleine BW. Sensitivity to instrumental contingency
degradation is mediated by the entorhinal cortex and its efferents via the
dorsal hippocampus. J Neurosci 2002;22:10976–84.
[64] Corbit LH, Muir JL, Balleine BW. Lesions of mediodorsal thalamus and anterior
thalamic nuclei produce dissociable effects on instrumental conditioning in
rats. Eur J Neurosci 2003;18:1286–94.
[65] Laubach M. Who’s on first? What’s on second? The time course of learning in
corticostriatal systems. Trends Neurosci 2005;28:509–11.
[66] Pasupathy A, Miller EK. Different time courses of learning-related activity in
the prefrontal cortex and striatum. Nature 2005;433:873–6.
[67] Ostlund SB, Balleine BW. Lesions of medial prefrontal cortex disrupt the
acquisition but not the expression of goal-directed learning. J Neurosci
2005;25:7763–70.
[68] Ding DC, Gabbott PL, Totterdell S. Differences in the laminar origin of projec-
tions from the medial prefrontal cortex to the nucleus accumbens shell and
core regions in the rat. Brain Res 2001;917:81–9.
[69] Berendse HW, Galis-de Graaf Y, Groenewegen HJ. Topographical organization
and relationship with ventral striatal compartments of prefrontal corticostri-
atal projections in the rat. J Comp Neurol 1992;316:314–47.
[70] Nauta WJH. Reciprocal links of the corpus striatum with the cerebral cor-
tex and limbic system: a common substrate for movement and thought?
In: Mueller Jonathon, editor. Neurology and psychiatry: a meeting of minds.
Basel: Karger; 1989. p. 43–63.
[71] Reep RL, Cheatwood JL, Corwin JV. The associative striatum: organization
of cortical projections to the dorsocentral striatum in rats. J Comp Neurol
2003;467:271–92.
[72] Kelley AE, Domesick VB, Nauta WJ. The amygdalostriatal projection in the
rat—an anatomical study by anterograde and retrograde tracing methods.
Neuroscience 1982;7:615–30.
[73] Balleine BW, Killcross AS, Dickinson A. The effect of lesions of the basolateral
amygdala on instrumental conditioning. J Neurosci 2003;23:666–75.
[74] Wang SH, Ostlund SB, Nader K, Balleine BW. Consolidation and reconsolida-
tion of incentive learning in the amygdale. J Neurosci 2005;25:830–5.
[75] Kawagoe R, Takikawa Y, Hikosaka O. Expectation of reward modules cognitive
signals in the basal ganglia. Nat Neurosci 1998;1:411–6.
[76] Hassani OK, Cromwell HC, Schultz W. Influence of expectation of different
rewards on behavior-related neuronal activity in the striatum. J Neurophysiol
2001;85:2477–89.
[77] Yin HH, Ostlund SB, Knowlton BJ, Balleine BW. The role of the dorsomedial
striatum in instrumental conditioning. Eur J Neurosci 2005;22:513–23.
[78] Kelley AE, Smith-Roe SL, Holahan MR. Response-reinforcement learning
is dependent on N-methyl-d-aspartate receptor activation in the nucleus
accumbens core. Proc Natl Acad Sci 1997;94:12174–9.
[79] Andrzejewski ME, Sadeghian K, Kelley AE. Central amygdalar and dorsal stri-
atal NMDA receptor involvement in instrumental learning and spontaneous
behavior. Behav Neurosci 2004;118.
[80] Yin HH, Knowlton BJ, Balleine BW. Blockade of NMDA receptors in the
dorsomedial striatum prevents action-outcome learning in instrumental con-
ditioning. Eur J Neurosci 2005;22:505–12.
[81] Lovinger DM, Partridge JG, Tang KC. Plastic control of striatal glutamatergic
transmission by ensemble actions of several neurotransmitters and targets
for drugs of abuse. Ann N Y Acad Sci 2003;1003:226–40.
[82] Tanaka SC, Balleine BW, O’Doherty JP. Calculating consequences: brain sys-
tems that encode the causal effects of actions. J Neurosci 2008;28:6750–
5.
[83] Alloway KD, Lou L, Nwabueze-Ogbo F, Chakrabarti S. Topography of corti-
cal projections to the dorsolateral neostriatum in rats: multiple overlapping
sensorimotor pathways. J Comp Neurol 2006;499:33–48.
[84] Ramanathan S, Hanley JJ, Deniau JM, Bolam JP. Synaptic convergence of motor
and somatosensory cortical afferents onto GABAergic interneurons in the rat
striatum. J Neurosci 2002;22:8158–69.
[85] Dickinson A. Instrumental conditioning. In: Mackintosh NJ, editor. Animal
cognition and learning. London: Academic Press; 1994. p. 4–79.
52 B.W. Balleine et al. / Behavioural Brain Research 199 (2009) 43–52
[86] Dickinson A, Wood N, Smith JW. Alcohol seeking by rats: action or habit? Q J
Exp Psychol B 2002;55:331–48.
[87] Robbins TW, Everitt BJ. Limbic-striatal memory systems and drug addiction.
Neurobiol Learn Mem 2002;78:625–36.
[88] Miles FJ, Everitt BJ, Dickinson A. Oral cocaine seeking by rats: action or habit?
Behav Neurosci 2003;117:927–38.
[89] Everitt BJ, Belin D, Economidou D, Pelloux Y, Dalley JW, Robbins TW.
Neural mechanisms underlying the vulnerability to develop compulsive
drug-seeking habits and addiction. Philos Trans R Soc Lond B Biol Sci
2008;363:3109–11.
[90] Adams CD. Variations in the sensitivity of instrumental responding to rein-
forcer devaluation. Q J Exp Psychol 1981;34B:77–98.
[91] Holman EW. Some conditions for the dissociation of consummatory and
instrumental behavior in rats. Learn Motiv 1975;6:358–66.
[92] Dickinson A, Nicholas DJ, Adams CD. The effect of the instrumental train-
ing contingency on susceptibility to reinforcer devaluation. Q J Exp Psychol
1983;35B:35–51.
[93] Yin HH, Knowlton BJ, Balleine BW. Inactivation of dorsolateral striatum
enhances sensitivity to changes in the action-outcome contingency in instru-
mental conditioning. Behav Brain Res 2006;166:189–96.
[94] Frankland PW, Wang Y, Rosner B, Shimizu T, Balleine BW, Dykens EM, et al.
Sensory-gating abnormalities in young males with fragile X syndrome and
Fmr1-knockout mice. Mol Psychiatry 2004;9:417–25.
[95] Jog MS, Kubota Y, Connolly CI, Hillegaart V, Graybiel AM. Building neural
representations of habits. Science 1999;286:1745–9.
[96] Barnes TD, Kubota Y, Hu D, Jin DZ, Graybiel AM. Activity of striatal neu-
rons reflects dynamic encoding and recoding of procedural memories. Nature
2005;437:1158–61.
[97] Costa RM, Cohen D, Nicolelis MA. Differential corticostriatal plasticity
during fast and slow motor skill learning in mice. Curr Biol 2004;14:
1124–34.
[98] Hernandez PJ, Schiltz CA, Kelley AE. Dynamic shifts in corticostriatal
expression patterns of the immediate early genes Homer 1a and Zif268 dur-
ing early and late phases of instrumental training. Learn Mem 2006;13:
599–608.
[99] Tang C, Pawlak AP, Prokopenko V, West MO. Changes in activity of the striatum
during formation of a motor habit. Eur J Neurosci 2007;25:1212–27.
[100] Costa RM, Lin SC, Sotnikova TD, Cyr M, Gainetdinov RR, Caron MG, et al. Rapid
alterations in corticostriatal ensemble coordination during acute dopamine-
dependent motor dysfunction. Neuron 2006;52:359–69.
[101] Yin HH, Knowlton BJ, Balleine BW. Lesions of dorsolateral striatum preserve
outcome expectancy but disrupt habit formation in instrumental learning.
Eur J Neurosci 2004;19:181–9.
[102] Featherstone RE, McDonald RJ. Dorsal striatum and stimulus-response
learning: lesions of the dorsolateral, but not dorsomedial, striatum
impair acquisition of a simple discrimination task. Behav Brain Res
2004;150(1–2):15–23.
[103] Featherstone RE, McDonald RJ. Lesions of the dorsolateral or dorsomedial
striatum impair performance of a previously acquired simple discrimination
task. Neurobiol Learn Mem 2005;84(3):159–67.
[104] Dickinson A, Balleine BW, Watt A, Gonzales F, Boakes RA. Overtraining
and the motivational control of instrumental action. Anim Learn Behav
1995;22:197–206.
[105] Glasner SV, Overmier JB, Balleine BW. The role of Pavlovian cues in alcohol
seeking in dependent and nondependent rats. J Stud Alcohol 2005;66:53–61.
[106] Nelson A, Killcross S. Amphetamine exposure enhances habit formation. J
Neurosci 2006;26:3805–12.
[107] Deroche-Gamonet V, Belin D, Piazza PV. Evidence for addiction-like behavior
in the rat. Science 2004;305:1014–7.
[108] Jedynak JP, Uslaner JM, Esteban JA, Robinson TE. Methamphetamine-induced
structural plasticity in the dorsal striatum. Eur J Neurosci 2007;25:847–
53.
[109] Alexander GE, DeLong MF, Strick PL. Parallel organisation of functionally
segregated circuits linking basal ganglia and cortex. Annu Rev Neurosci
1986;9:357–81.
[110] Kelly RM, Strick PL. Macro-architecture of basal ganglia loops with the cerebral
cortex: use of rabies virus to reveal multisynaptic circuits. Prog Brain Res
2004;143.
[111] Morgane PJ, Galler JR, Mokler DJ. A review of systems and networks of the
limbic forebrain/limbic midbrain. Prog Neurobiol 2005;75:143–60.
[112] Opris I, Bruce CJ. Neural circuitry of judgment and decision mechanisms. Brain
Res Brain Res Rev 2005;48:509–26.
[113] Volz KG, Schubotz RI, von Cramon DY. Decision-making and the frontal lobes.
Curr Opin Neurol 2006;19:401–6.
[114] Goldman-Rakic PS. Architecture of the prefrontal cortex and the central exec-
utive. Ann N Y Acad Sci 1995;769:71–83.
[115] Fuster JM. Executive frontal functions. Exp Brain Res 2000;133:66–70.
[116] Frank MJ, Loughry B, O’Reilly RC. Interactions between frontal cortex and
basal ganglia in working memory: a computational model. Cogn Affect Behav
Neurosci 2001;1:137–60.
[117] Frank MJ, Claus ED. Anatomy of a decision: striato-orbitofrontal interac-
tions in reinforcement learning, decision making, and reversal. Psychol Rev
2006;113:300–26.
[118] Kemp JM, Powell TPS. The connections of the striatum and globus pallidus:
synthesis and speculation. Philos Trans R Soc London Ser B 1971;262:441–57.
[119] Miyachi S, Hasegawa YT, Gerfen CR. Coincident stimulation of convergent
cortical inputs enhances immediate early gene induction in the striatum.
Neuroscience 2005;134:1013–22.
[120] Ostlund SB, Balleine BW. The contribution of orbitofrontal cortex to action
selection. Ann N Y Acad Sci 2007;1121:174–92.
[121] Bailey KR, Mair RG. Effects of frontal cortex lesions on action sequence learning
in the rat. Eur J Neurosci 2007;25:2905–15.
[122] Nakano K, Kayahara T, Tsutsumi T, Ushiro H. Neural circuits and functional
organization of the striatum. J Neurol 2000;247:1–15.
[123] Haber SN, Fudge JL, McFarland NR. Striatonigrostriatal pathways in primates
form an ascending spiral from the shell to the dorsolateral striatum. J Neurosci
2000;20:2369–82.
[124] Zahm DS. An integrative neuroanatomical perspective on some subcortical
substrates of adaptive responding with emphasis on the nucleus accumbens.
Neurosci Biobehav Rev 2000;24:85–105.
[125] Zackheim J, Abercrombie ED. Thalamic regulation of striatal acetylcholine
efflux is both direct and indirect and qualitatively altered in the dopamine-
depleted striatum. Neuroscience 2005;131:423–36.
[126] Nadjar A, Brotchie JM, Guigoni C, Li Q, Zhou SB, Wang GJ, et al. Phenotype of
striatofugal medium spiny neurons in Parkinsonian and dyskinetic nonhuman
primates: a call for a reappraisal of the functional organization of the basal
ganglia. J Neurosci 2006;26:8653–61.
[127] Sadek AR, Magill PJ, Bolam JP. A single-cell analysis of intrinsic connectivity
in the rat globus pallidus. J Neurosci 2007;27:6352–62.
[128] Reynolds JN, Hyland BI, Wickens JR. A cellular mechanism of reward-related
learning. Nature 2001;413:67–70.
[129] Faure A, Haberland U, Condé F, El Massioui N. Lesion to the nigrostriatal
dopamine system disrupts stimulus-response habit formation. J Neurosci
2005;25:2771–80.
[130] Gonzales C, Chesselet MF. Amygdalonigral pathway: an anterograde study
in the rat with Phaseolus vulgaris leucoagglutinin (PHA-L). J Comp Neurol
1990;297:182–200.
[131] Holland PC, Gallagher M. Double dissociation of the effects of lesions of baso-
lateral and central amygdala on conditioned stimulus-potentiated feeding
and Pavlovian-instrumental transfer. Eur J Neurosci 2003;17:1680–94.
[132] Corbit LH, Balleine BW. Double dissociation of basolateral and central
amygdala lesions on the general and outcome-specific forms of Pavlovian-
instrumental transfer. J Neurosci 2005;25:962–70.
[133] El-Amamy H, Holland PC. Substantia nigra pars compacta is critical to both
the acquisition and expression of learned orienting of rats. Eur J Neurosci
2006;24:270–6.
[134] El-Amamy H, Holland PC. Dissociable effects of disconnecting amygdala cen-
tral nucleus from the ventral tegmental area or substantia nigra on learned
orienting and incentive motivation. Eur J Neurosci 2007;25:1557–67.
[135] Calabresi P, Picconi B, Tozzi A, Di Filippo M. Dopamine-mediated regulation
of corticostriatal synaptic plasticity. Trends Neurosci 2007;30:211–9.
[136] Haber SN. The primate basal ganglia: parallel and integrative networks. J Chem
Neuroanat 2003;26:317–30.
[137] Han S, Holland PC, Gallagher M. Disconnection of the amygdala central
nucleus and substantia innominata/nucleus basalis disrupts increments
in conditioned stimulus processing in rats. Behav Neurosci 1999;113:
143–51.
[138] Pisa M. Motor functions of the striatum in the rat: critical role of the lateral
region in tongue and forelimb reaching. Neuroscience 1998;24(2):453–63.
[139] Squire LR, Zola-Morgan S. Memory: brain systems and behavior. Trends Neu-
rosci 1988;11:170–5.
[140] de Wit S, Kosaki Y, Balleine BW, Dickinson A. Dorsomedial prefrontal cortex
resolves response conflict in rats. J Neurosci 2006;26:5224–9.
[141] Balleine BW, Dickinson A. Effect of lesions of the insular cortex on instru-
mental conditioning: Evidence for a role in incentive memory. J Neurosci
2000;20:8954–64.
[142] Cardinal RN, Everitt BJ. Neural and psychological mechanisms underly-
ing appetitive learning: links to drug addiction. Curr Opin Neurobiol
2004;14:156–62.
[143] de Wit S, Kosaki Y, Balleine BW, Dickinson A. Dorsomedial prefrontal cortex
resolves response conflict in rats. J Neurosci 2006;26:5224–9.
[144] Cardinal RN, Parkinson JA, Hall J, Everitt BJ. Emotion and motivation: the role
of the amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav
Rev 2002;26:321–52.