Flexible Neural Representations of Value in the Primate Brain
C. DANIEL SALZMAN
Departments of Neuroscience and Psychiatry, Columbia University, New York State Psychiatric Institute, New York, New York 10032, USA
Search for more papers by this authorJOSEPH J. PATON
Departments of Neuroscience and Psychiatry, Columbia University, New York State Psychiatric Institute, New York, New York 10032, USA
Search for more papers by this authorMARINA A. BELOVA
Departments of Neuroscience and Psychiatry, Columbia University, New York State Psychiatric Institute, New York, New York 10032, USA
Search for more papers by this authorSARA E. MORRISON
Departments of Neuroscience and Psychiatry, Columbia University, New York State Psychiatric Institute, New York, New York 10032, USA
Search for more papers by this authorC. DANIEL SALZMAN
Departments of Neuroscience and Psychiatry, Columbia University, New York State Psychiatric Institute, New York, New York 10032, USA
Search for more papers by this authorJOSEPH J. PATON
Departments of Neuroscience and Psychiatry, Columbia University, New York State Psychiatric Institute, New York, New York 10032, USA
Search for more papers by this authorMARINA A. BELOVA
Departments of Neuroscience and Psychiatry, Columbia University, New York State Psychiatric Institute, New York, New York 10032, USA
Search for more papers by this authorSARA E. MORRISON
Departments of Neuroscience and Psychiatry, Columbia University, New York State Psychiatric Institute, New York, New York 10032, USA
Search for more papers by this authorAbstract
Abstract: The amygdala and orbitofrontal cortex (OFC) are often thought of as components of a neural circuit that assigns affective significance—or value—to sensory stimuli so as to anticipate future events and adjust behavioral and physiological responses. Much recent work has been aimed at understanding the distinct contributions of the amygdala and OFC to these processes, but a detailed understanding of the physiological mechanisms underlying learning about value remains lacking. To gain insight into these processes, we have focused initially on characterizing the neural signals of the primate amygdala, and more recently of the primate OFC, during appetitive and aversive reinforcement learning procedures. We have employed a classical conditioning procedure whereby monkeys form associations between visual stimuli and rewards or aversive stimuli. After learning these initial associations, we reverse the stimulus-reinforcement contingencies, and monkeys learn these new associations. We have discovered that separate populations of neurons in the amygdala represent the positive and negative value of conditioned visual stimuli. This representation of value updates rapidly upon image value reversal, as fast as monkeys learn, often within a single trial. We suggest that representations of value in the amygdala may change through multiple interrelated mechanisms: some that arise from fairly simple Hebbian processes, and others that may involve gated inputs from other brain areas, such as the OFC.
REFERENCES
- 1 Ochsner, K.N. & J.J. Gross. 2005. The cognitive control of emotion. Trends Cogn. Sci. 9: 242–249.
- 2 Murray, E.A. 2008. Neuropsychology of primate reward processes. In New Encyclopedia of Neuroscience. L.R. Squire Ed. Elsevier. In press.
- 3 Pennartz, C.M. 1997. Reinforcement learning by Hebbian synapses with adaptive thresholds. Neuroscience 81: 303–319.
- 4 Rescorla, R.A. & A.R. Wagner. 1972. A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and non-reinforcement. In Classical Conditioning II: Current Research and Theory. A.H. Black & W.F. Prokasy Eds.: 64–99. Appleton Century Crofts. New York .
- 5 Sutton, R. & A. Barto. 1998. Reinforcement Learning. MIT Press. Cambridge , MA .
- 6 Schultz, W., P. Dayan & P.R. Montague. 1997. A neural substrate of prediction and reward. Science 275: 1593–1599.
- 7 Pearce, J. & G. Hall. 1980. A model for Pavlovian conditioning: variations in the effectiveness of conditioned but not unconditioned stimuli. Psychol. Rev. 87: 532–552.
- 8 LeDoux, J.E. 2000. Emotion circuits in the brain. Annu. Rev. Neurosci. 23: 155–184.
- 9 Everitt, B.J., R.N. Cardinal, J.A. Parkinson & T.W. Robbins. 2003. Appetitive behavior: impact of amygdala-dependent mechanisms of emotional learning. Ann. N. Y. Acad. Sci. 985: 233–250.
- 10 Baxter, M. & Murray, EA. 2002. The amygdala and reward. Nat. Rev. Neurosci. 3: 563–573.
- 11 Amaral, D., J. Price, A. Pitkanen & S. Carmichael. 1992. Anatomical organization of the primate amygdaloid complex. In The Amygdala: Neurobiological Aspects of Emotion, Memory, and Mental Dysfunction. J. Aggleton Ed.: 1–66. Wiley–Liss. New York .
- 12 Swanson, L.W. & G.D. Petrovich. 1998. What is the amygdala? Trends Neurosci. 21: 323–331.
- 13 Ghashghaei, H.T. & H. Barbas. 2002. Pathways for emotion: interactions of prefrontal and anterior temporal pathways in the amygdala of the rhesus monkey. Neuroscience 115: 1261–1279.
- 14 Stefanacci, L. & D.G. Amaral. 2002. Some observations on cortical inputs to the macaque monkey amygdala: an anterograde tracing study. J. Comp. Neurol. 451: 301–323.
- 15 Repa, J.C. et al . 2001. Two different lateral amygdala cell populations contribute to the initiation and storage of memory. Nat. Neurosci. 4: 724–731.
- 16 Schoenbaum, G., A.A. Chiba & M. Gallagher. 1999. Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning. J. Neurosci. 19: 1876–1884.
- 17 Pascoe, J.P. & B.S. Kapp. 1985. Electrophysiological characteristics of amygdaloid central nucleus neurons during Pavlovian fear conditioning in the rabbit. Behav. Brain Res. 16: 117–133.
- 18 Quirk, G.J., C. Repa & LeDoux, J.E. 1995. Fear conditioning enhances short-latency auditory responses of lateral amygdala neurons: parallel recordings in the freely behaving rat. Neuron 15: 1029–1039.
- 19 Applegate, C.D., R.C. Frysinger, B.S. Kapp & M. Gallagher. 1982. Multiple unit activity recorded from amygdala central nucleus during Pavlovian heart rate conditioning in rabbit. Brain Res. 238: 457–462.
- 20 Goosens, K.A., J.A. Hobin & S. Maren. 2003. Auditory-evoked spike firing in the lateral amygdala and Pavlovian fear conditioning: mnemonic code or fear bias? Neuron 40: 1013–1022.
- 21 Collins, D.R. & D. Pare. 2000. Differential fear conditioning induces reciprocal changes in the sensory responses of lateral amygdala neurons to the CS(+) and CS(-). Learn. Mem. 7: 97–103.
- 22 Maren, S. & G.J. Quirk. 2004. Neuronal signaling of fear memory. Nat. Rev. Neurosci. 5: 844–852.
- 23 Schoenbaum, G., A.A. Chiba & M. Gallagher. 1998. Orbitofrontal cortex and basolateral amygdala encode expected outcomes during learning. Nat. Neurosci. 1: 155–159.
- 24 Maren, S. 2000. Auditory fear conditioning increases CS-elicited spike firing in lateral amygdala neurons even after extensive overtraining. Eur. J. Neurosci. 12: 4047–4054.
- 25 Rorick-Kehn, L.M. & J.E. Steinmetz. 2005. Amygdalar unit activity during three learning tasks: eyeblink classical conditioning, Pavlovian fear conditioning, and signaled avoidance conditioning. Behav. Neurosci. 119: 1254–1276.
- 26 Sugase-Miyamoto, Y. & B.J. Richmond. 2005. Neuronal signals in the monkey basolateral amygdala during reward schedules. J. Neurosci. 25: 11071–11083.
- 27 Dehaene, S. & J.P. Changeux. 2000. Reward-dependent learning in neuronal networks for planning and decision making. Prog. Brain Res. 126: 217–229.
- 28 Sanghera, M.K., E.T. Rolls & A. Roper-Hall. 1979. Visual responses of neurons in the dorsolateral amygdala of the alert monkey. Exp. Neurol. 63: 610–626.
- 29 Nishijo, H., T. Ono & H. Nishino. 1988. Single neuron responses in amygdala of alert monkey during complex sensory stimulation with affective significance. J. Neurosci. 8: 3570–3583.
- 30 Nakamura, K., A. Mikami & K. Kubota. 1992. Activity of single neurons in the monkey amygdala during performance of a visual discrimination task. J. Neurophysiol. 67: 1447–1663.
- 31 Wilson, F.A. & E.T. Rolls. 2005. The primate amygdala and reinforcement: a dissociation between rule-based and associatively-mediated memory revealed in neuronal activity. Neuroscience 133: 1061–1072.
- 32 Rolls, E. 1992. Neurophysiology and functions of the primate amygdala. In The Amygdala: Neurobiological Aspects of Emotion, Memory, and Mental Dysfunction. J. Aggleton, Ed.: 143–166. Wiley–Liss. New York .
- 33 Paton, J., M. Belova, S. Morrison & C. Salzman. 2006. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439: 865–870.
- 34 Haber, S.N., K. Kunishio, M. Mizobuchi & E. Lynd-Balta. 1995. The orbital and medial prefrontal circuit through the primate basal ganglia. J. Neurosci. 15: 4851–4867.
- 35 Cavada, C., T. Company, J. Tejedor, et al . 2000. The anatomical connections of the macaque monkey orbitofrontal cortex: a review. Cereb. Cortex 10: 220–242.
- 36 Carmichael, S.T. & J.L. Price. 1995. Limbic connections of the orbital and medial prefrontal cortex in macaque monkeys. J. Comp. Neurol. 363: 615–641.
- 37
Stefanacci, L. &
D.G. Amaral. 2000. Topographic organization of cortical inputs to the lateral nucleus of the macaque monkey amygdala: a retrograde tracing study.
J. Comp. Neurol. 421: 52–79.
10.1002/(SICI)1096-9861(20000522)421:1<52::AID-CNE4>3.0.CO;2-O CASPubMedWeb of Science®Google Scholar
- 38 Bechara, A., H. Damasio & A.R. Damasio. 2000. Emotion, decision making and the orbitofrontal cortex. Cereb. Cortex 10: 295–307.
- 39 Bechara, A., H. Damasio & A.R. Damasio. 2003. Role of the amygdala in decision-making. Ann. N.Y. Acad. Sci. 985: 356–369.
- 40 Bechara, A., D. Tranel, H. Damasio & A.R. Damasio. 1996. Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex. Cereb. Cortex 6: 215–225.
- 41 Chow, T.W. 2000. Personality in frontal lobe disorders. Curr. Psychiatry Rep. 2: 446–451.
- 42 Izquierdo, A., R.K. Suda & E.A. Murray. 2004. Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. J. Neurosci. 24: 7540–7548.
- 43 Baxter, M., A. Parker, C.C. Lindner, et al . 2000. Control of response selection by reinforcer value requires interaction of amygdala and orbital prefrontal cortex. J. Neurosci. 20: 4311–4319.
- 44 Cox, S.M., A. Andrade & I.S. Johnsrude. 2005. Learning to like: a role for human orbitofrontal cortex in conditioned reward. J. Neurosci. 25: 2733–2740.
- 45 Ursu, S. & C.S. Carter. 2005. Outcome representations, counterfactual comparisons and the human orbitofrontal cortex: implications for neuroimaging studies of decision-making. Cogn. Brain Res. 23: 51–60.
- 46 O’Doherty, J., M.L. Kringelbach, E.T. Rolls, et al . 2001. Abstract reward and punishment representations in the human orbitofrontal cortex. Nature Neurosci. 4: 95–102.
- 47 Thorpe, S.J., E.T. Rolls & S. Maddison. 1983. The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp. Brain Res. 49: 93–115.
- 48 Schoenbaum, G., A. Chiba & M. Gallagher. 1998. Orbitalfrontal cortex and basolateral amygdala encode expected outcomes during learning. Nature Neurosci. 1: 155–159.
- 49 Schoenbaum, G., A.A. Chiba & M. Gallagher. 1999. Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning. J. Neurosci. 19: 1876–1884.
- 50 Schoenbaum, G., B. Setlow, M.P. Saddoris & M. Gallagher. 2003. Encoding predicted outcome and acquired value in orbitofrontal cortex during cue sampling depends upon input from basolateral amygdala. Neuron 39: 855–867.
- 51 Rolls, E.T., H.D. Critchley, R. Mason & E.A. Wakeman. 1996. Orbitofrontal cortex neurons: role in olfactory and visual association learning. J. Neurophysiol. 75: 1970–1981.
- 52 Tremblay, L. & W. Schultz. 1999. Relative reward preference in primate orbitofrontal cortex. Nature 398: 704–708.
- 53 Wallis, J.D. & E.K. Miller. 2003. Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task. Eur. J. Neurosci. 18: 2069–2081.
- 54 Tremblay, L. & W. Schultz. 2000. Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. J. Neurophysiol. 83: 1877–1885.
- 55 Roesch, M.R. & C.R. Olson. 2004. Neuronal activity related to reward value and motivation in primate frontal cortex. Science 304: 307–310.
- 56 Padoa-Schioppa, C. & J.A. Assad. 2006. Neurons in the orbitofrontal cortex encode economic value. Nature 441: 223–226.
- 57 Feierstein, C.E., M.C. Quirk, N. Uchida, et al . 2006. Representation of spatial goals in rat orbitofrontal cortex. Neuron 51: 495–507.
- 58 Roesch, M.R., A.R. Taylor & G. Schoenbaum. 2006. Encoding of time-discounted rewards in orbitofrontal cortex is independent of value representation. Neuron 51: 509–520.
- 59 Ostlund, S.B. & B.W. Balleine. 2007. Orbitofrontal cortex mediates outcome encoding in Pavlovian but not instrumental conditioning. J. Neurosci. 27: 4819–4825.
- 60 Mazur, J.E. 2006. Learning and Behavior. Prentice Hall. Upper Saddle River , NJ .
- 61
Pavlov, I.P.
1927. Conditioned Reflexes. Oxford University Press.
London
.
10.3109/00016482709120085 Google Scholar
- 62 Lang, P.J., M.M. Bradley & B.N. Cuthbert. 1990. Emotion, attention, and the startle reflex. Psychol. Rev. 97: 377–395.
- 63 Gallistel, C.R., S. Fairhurst & P. Balsam. 2004. The learning curve: implications of a quantitative analysis. Proc. Natl. Acad. Sci. USA 101: 13124–13131.
- 64 Belova, M.A., J.J. Paton, S.E. Morrison & C.D. Salzman. 2007. Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala. Neuron. 55(6): 970–984.
- 65 Saddoris, M.P., M. Gallagher & G. Schoenbaum. 2005. Rapid associative encoding in basolateral amygdala depends on connections with orbitofrontal cortex. Neuron 46: 321–331.
- 66 Green, D.M. & J.A. Swets. 1966. Signal Detection Theory and Psychophysics. Wiley. New York .
- 67 Morrison, S.E. & C.D. Salzman. 2007. Primate orbitofrontal cortex and amygdala encode stimulus value with a similar time course during reinforcement learning. Soc. Neurosci. Abstr. 934.4.
- 68 Rolls, E. 2000. Neurophysiology and functions of the primate amygdala, and the neural basis of emotion. In The Amygdala: a Functional Analysis. J. Aggleton, Ed.: 447–478. Oxford University Press. New York .
- 69 Fellows, L.K. & M.J. Farah. 2003. Ventromedial frontal cortex mediates affective shifting in humans: evidence from a reversal learning paradigm. Brain 126: 1830–1837.
- 70 Izquierdo, A. & E.A. Murray. 2007. Selective bilateral amygdala lesions in rhesus monkeys fail to disrupt object reversal learning. J. Neurosci. 27: 1054–1062.
- 71 Schoenbaum, G., B. Setlow, S.L. Nugent, M.P. Saddoris & M. Gallagher. 2003. Lesions of orbitofrontal cortex and basolateral amygdala complex disrupt acquisition of odor-guided discriminations and reversals. Learn. Mem. 10: 129–140.
- 72 Swinehart, C.D. & L.F. Abbott. 2005. Supervised learning through neuronal response modulation. Neural Comput. 17: 609–631.
- 73 Swinehart, C.D., K. Bouchard, P. Partensky & L.F. Abbott. 2004. Control of network activity through neuronal response modulation. Neurocomputing 58–60: 327–335.