We commence by examining the standard methods for identifying action-value neurons using a simulation of an operant learning experiment. We simulated a task, in which the subject repeatedly chooses between two alternative actions, which yield a binary reward with a probability that depends on the action. Specifically, each session in the simulation was composed of four blocks such that the probabilities of rewards were fixed within a block and varied between the blocks. The probabilities of reward in the blocks were (0.1,0.5), (0.9,0.5), (0.5,0.9) and (0.5,0.1) for actions 1 and 2, respectively (Figure 1A). The order of blocks was random and a block terminated when the more rewarding action was chosen more than 14 times within 20 consecutive trials (Ito and Doya, 2015a; Samejima et al., 2005).

Figure 1

Download asset Open asset

To simulate learning behavior, we used the Q-learning framework (Equations 1 and 2 with $\alpha =0.1$ and $\beta =2.5$ (taken from distributions reported in [Kim et al., 2009]) and initial conditions $Q_i\left(1\right)=0.5$). As demonstrated in Figure 1A, the model learned: the probability of choosing the more rewarding alternative increased over trials (black line). To model the action-value neurons, we simulated neurons whose firing rate is a linear function of one of the two Q-values and whose spike count in a 1 sec trial is randomly drawn from a corresponding Poisson distribution (see Materials and methods). The firing rates and spike counts of two such neurons, representing action-values 1 and 2, are depicted in Figure 1B in red and blue, respectively.

One standard method for identifying action-value neurons is to compare neurons' spike counts after learning, at the end of the blocks (horizontal bars in Figure 1B). Considering the red-labeled Poisson neuron, the spike count in the last 20 trials of the second block, in which the probability of reward associated with action 1 was 0.9, was significantly higher than that count in the first block, in which the probability of reward associated with action 1 was 0.1 (p<0.01; rank sum test). By contrast, there was no significant difference in the spike counts between the third and fourth blocks, in which the probability of reward associated with action 1 was equal (p=0.91; rank sum test). This is consistent with the fact that the red-labeled neuron was an action 1-value neuron: its firing rate was a linear function of the value of action 1 (Figure 1B, red) Similarly for the blue labeled neuron, the spike counts in the last 20 trials of the first two blocks were not significantly different (p=0.92; rank sum test), but there was a significant difference in the counts between the third and fourth blocks (p<0.001; rank sum test). These results are consistent with the probabilities of reward associated with action 2 and the fact that in our simulations, this neuron’s firing rate was modulated by the value of action 2 (Figure 1B, blue).

This approach for identifying action-value neurons is limited, however, for several reasons. First, it considers only a fraction of the data, the last 20 trials in a block. Second, action-value neurons are not expected to represent the block average probabilities of reward. Rather, they will represent a subjective estimate, which is based on the subject-specific history of actions and rewards. Therefore, it is more common to identify action-value neurons by regressing the spike count on subjective action-values, estimated from the subject’s history of choices and rewards (Funamizu et al., 2015; Ito and Doya, 2015a; Ito and Doya, 2015b; Kim et al., 2009; Lau and Glimcher, 2008; Samejima et al., 2005). Note that when studying behavior in experiments, we have no direct access to these estimated action-values, in particular because the values of the parameters $\alpha$ and $\beta$ are unknown. Therefore, following common practice, we estimated the values of $\alpha$ and $\beta$ from the model’s sequence of choices and rewards using maximum likelihood, and used the estimated learning rate ($\alpha$) and the choices and rewards to estimate the action-values (thin lines in Figure 1C, see Materials and methods). These estimates were similar to the true action-value, which underlay the model’s choice behavior (thick lines in Figure 1C).

Next, we regressed the spike count of each simulated neuron on the two estimated action-values from its corresponding session. As expected, the t-value of the regression coefficient of the red-labeled action 1-value neuron was significant for the estimated $Q_1$ $\left(t_{182}\left(Q_1\right)=4.05\right)$ but not for the estimated $Q_2$ $\left(t_{182}\left(Q_2\right)=-0.27\right)$. Similarly, the t-value of the regression coefficient of the blue-labeled action 2-value neuron was significant for the estimated $Q_2$ $\left(t_{182}\left(Q_2\right)=3.05\right)$ but not for the estimated $Q_1$ $\left(t_{182}\left(Q_1\right)=0.78\right)$.

A population analysis of the t-values of the two regression coefficients is depicted in Figure 1D,E. As expected, a substantial fraction (42%) of the simulated neurons were identified as action-value neurons. Only 2% of the simulated neurons had significant regression coefficients with both action-values. Such neurons are typically classified as state $\left(\mathrm{\Sigma }Q\right)$ or policy (also known as preference) $\left(\mathrm{\Delta }Q\right)$ neurons, if the two regression coefficients have the same or different signs, respectively (Ito and Doya, 2015a). Note that despite the fact that by construction, all neurons were action-value neurons, not all of them were detected as such by this method. This failure occurred for two reasons. First, the estimated action-values are not identical to the true action-values, which determine the firing rates. This is because of the finite number of trials and the stochasticity of choice (note the difference, albeit small, between the thin and thick lines in Figure 1C). Second and more importantly, the spike count in a trial is only a noisy estimate of the firing rate because of the Poisson generation of spikes.

Several prominent studies have implemented the methods we described in this section and reported that a substantial fraction (10–40% depending on significance threshold) of striatal neurons represent action-values (Ito and Doya, 2015a; Ito and Doya, 2015b; Samejima et al., 2005). In the next two sections we show that these methods, and similar methods employed by other studies (Cai et al., 2011; FitzGerald et al., 2012; Funamizu et al., 2015; Guitart-Masip et al., 2012; Her et al., 2016; Ito and Doya, 2009; Kim et al., 2013; Kim et al., 2009; Kim et al., 2012; 2007; Lau and Glimcher, 2008; Stalnaker et al., 2010; Wang et al., 2013; Wunderlich et al., 2009) are all subject to at least one of two major confounds.

In the previous sections we demonstrated that irrelevant temporal correlations may lead to the erroneous classification of neurons as representing action-values, even if their activity is task-independent. Here we address an unrelated confound. We show that neurons that encode different decision variables, in particular policy, may be erroneously classified as representing action-values. For clarity, we will commence by discussing this caveat independently of the temporal correlations confound. Specifically, we show that neurons whose firing rate encodes the policy (probability of choice) may be erroneously classified as representing action-values, even when this policy emerged in the absence of any implicit or explicit action-value representation. We will conclude by discussing a possible solution that addresses this and the temporal correlations confounds.

Policy without action-value representation

It is well-known that operant learning can occur in the absence of any value computation, for example, as a result of direct-policy learning (Mongillo et al., 2014). Several studies have shown that reward-modulated synaptic plasticity can implement direct-policy reinforcement learning (Darshan et al., 2014; Fiete et al., 2007; Frémaux et al., 2010; Loewenstein, 2008; Loewenstein, 2010; Loewenstein and Seung, 2006; Neiman and Loewenstein, 2013; Seung, 2003; Urbanczik and Senn, 2009).

For concreteness, we consider a particular reinforcement learning algorithm, in which the probability of choice $\mathrm{P}\mathrm{r}\left(a\left(t\right)=1\right)$ is determined by a single variable $W$ that is learned in accordance with the REINFORCE learning algorithm (Williams, 1992): $\mathrm{P}\mathrm{r}\left(a\left(t\right)=1\right)=\frac{1}{1+e^{-W\left(t\right)}}$ where $∆W\left(t\right)=\alpha \bullet (2\bullet R\left(t\right)-1)\bullet (a\left(t\right)-\mathrm{P}\mathrm{r}\left(a\left(t\right)=1\right))$, where $\alpha$ is the learning rate, $R\left(t\right)$ is the binary reward in trial $t$ and $a\left(t\right)$ is a binary variable indicating whether action 1 was chosen in trial $t$. In our simulations $W\left(t=1\right)=0$, $\alpha =0.17$. For biological implementation of this algorithm see (Loewenstein, 2010; Seung, 2003).

We tested this model in the experimental design of Figure 1 (Figure 5A). As expected, the model learned to prefer the action associated with a higher probability of reward, completing the four blocks within 228 trials on average (standard deviation 62 trials).

Figure 5 with 1 supplement see all

Download asset Open asset

A possible solution to the policy confound

The policy confound emerged because policy and action-values are correlated. To distinguish between the two possible representations, we should seek a variable that is correlated with the action-value but uncorrelated with the policy. Consider the sum of the two action-values. It is easy to see that $\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(Q_1+Q_2,Q_1-Q_2\right)\propto \mathrm{V}\mathrm{a}\mathrm{r}\left(Q_1\right)-\mathrm{V}\mathrm{a}\mathrm{r}\left(Q_2\right)$. Therefore, if the variances of the two action-values are equal, their sum is uncorrelated with their difference. An action-value neuron is expected to be correlated with the sum of action-values. By contrast, a policy neuron, modulated by the difference in action-values is expected to be uncorrelated with this sum.

We repeated the simulations of Figure 4 (which addresses the temporal correlations confound), considering three types of neurons: action-value neurons (of Figure 1), random-walk neurons (of Figure 2), and policy neurons (of Figure 5). As in Figure 4, we considered the spike counts of the three types of neurons in the last 200 trials of the session, but now we regressed them on the sum of reward probabilities (state; in this experimental design the reward probabilities are also the objective action-values, which the subject learns). We found that only 4.5 and 6% of the random-walk and policy neurons, respectively, were significantly correlated with the sum of reward probabilities (5% chance level). By contrast, 47% of the action-value neurons were significantly correlated with this sum.

This method is able to distinguish between policy and action-value representations. However, it will fail in the case of state representation because both state and action-values are correlated with the sum of probabilities of reward. To dissociate between state and action-value representations, we can consider the difference in reward probabilities because this difference is correlated with the action-values but is uncorrelated with the state. Regressing the spike count on both the sum and difference of the probabilities of reward, a random-walk neuron is expected to be correlated with none, a policy neuron is expected to be correlated only with the difference, whereas an action-value neuron is expected to be correlated with both (this analysis is inspired by Fig. S8b in (Wang et al., 2013) in which the predictors in the regression model were policy and state). We now classify a neuron that passes both significance tests as an action-value neuron. Indeed, for a significance threshold of p<0.05 (for each test), only 0.2% of the random-walk neurons and 5% of the policy neurons were classified as action-value neurons. By contrast, 32% of the action-value neurons were classified as such (Figure 6). Note that in this analysis only when more than 5% of the neurons are classified as action-value neurons we have support for the hypothesis that there is action-value representation rather than policy or state representation.

Figure 6

Download asset Open asset

A word of caution is that the analysis should be performed only after the learning converges. This is because stochastic fluctuations in the learning process may be reflected in the activities of neurons representing decision-related variables. As a result, policy or state-representing neurons may appear correlated with the orthogonal variables. For the same reason, any block-related heterogeneity in neural activity could also result in this confound (O'Doherty, 2014).

To conclude, it is worthwhile repeating the key features of the analysis proposed in this section:

1. Trial design is necessary because otherwise temporal correlations in spike count may inflate the fraction of neurons that pass the significance tests.

2. Regression should be performed on reward probabilities (i.e., the objective action-values) and not on estimated action-values. The reason is that because the estimated action-values evolve over time, this trial design does not eliminate all temporal correlations between them (Figure 2—figure supplement 9).

3. Reward probabilities associated with the two actions should be chosen such that their variances should be equal. Otherwise policy or state neurons may be erroneously classified as action-value neurons.

It is well known in statistics that the regression coefficient between two independent slowly-changing variables is on average larger (in absolute value) than this coefficient when the series are devoid of a temporal structure. If these temporal correlations are overlooked, the probability of a false-positive is underestimated (Granger and Newbold, 1974). When searching for action-value representation in a block design, then by construction, there are positive correlations in the predictor (action-values). Positive temporal correlations in the dependent variable (neural activity) will result in an inflation of the false-positive observations, compared with the naïve expectation.

This confound occurs only when there are temporal correlations in both the predictor and the dependent variable. In a trial design, in which the predictor is chosen independently in each trial and thus has no temporal structure, we do not expect this confound. However, when studying incremental learning, it is difficult to randomize the predictor in each trial, making the task of identifying neural correlates of learning, and specifically action-values, challenging. With respect to the dependent variable (neural activity), temporal correlations in BOLD signal and their consequences have been discussed (Arbabshirani et al., 2014; Woolrich et al., 2001). Considering electrophysiological recordings, there have been attempts to remove these correlations, for example, using previous spike counts as predictors (Kim et al., 2013). However, these are not sufficient because they are unable to remove all task-independent temporal correlations (see also Figure 2—figure supplements 4–10). When repeating these analyses, we erroneously classified a fraction of neurons as representing action-value that is comparable to that reported in the striatum. The probability of a false-positive identification of a neuron as representing action-value depends on the magnitude and type of temporal correlations in the neural activity. Therefore, we cannot predict the fraction of erroneously classified neurons expected in various experimental settings and brain areas.

One may argue that the fact that action-value representations are reported mostly in a specific brain area, namely the striatum, is an indication that their identification there is not a result of the temporal correlations confound. However, because different brain regions are characterized by different spiking statistics, we expect different levels of erroneous identification of action-value neurons in different parts of the brain and in different experimental settings. Indeed, the fraction of erroneously identified action-value neurons differed between the auditory and motor cortices (compare B and D within Figure 2—figure supplement 2). Furthermore, many studies reported action-value representation outside of the striatum, in brain areas including the supplementary motor area and presupplementary eye fields (Wunderlich et al., 2009), the substantia nigra/ventral tegmental area (Guitart-Masip et al., 2012) and ventromedial prefrontal cortex, insula and thalamus (FitzGerald et al., 2012).

Considering the ventral striatum, our analysis on recordings from (Ito and Doya, 2009) indicates that the identification of action-value representations there may have been erroneous, resulting from temporally correlated firing rates (Figure 3 and Figure 2—figure supplement 3). It should be noted that the fraction of action-value neurons reported in (Ito and Doya, 2009) is low relative to other publications, a difference that has been attributed to the location of the recording in the striatum (ventral as opposed to dorsal). It would be interesting to apply this method to other striatal recordings (Ito and Doya, 2015a; Samejima et al., 2005; Wang et al., 2013). We were unable to directly analyze these recordings from the dorsal striatum because relevant raw data is not publicly available. However, previous studies have reported that the firing rates of dorsal-striatal neurons change slowly over time (Gouvêa et al., 2015; Mello et al., 2015). As a result, identification of apparent action-value representation in dorsal-striatal neurons may also be the result of this confound.

Temporal correlations naturally emerge in experiments composed of multiple trials. Participants become satiated, bored, tired, etc., which may affect neuronal activity. In particular, learning in operant tasks is associated, by construction, with variables that are temporally correlated. If neural activity is correlated with performance (e.g., accumulated rewards in the last several trials) then it is expected to have temporal correlations, which may lead to an erroneous classification of the neurons as representing action-values.

Action-values are not the only example of slowly-changing variables. Any variable associated with incremental learning, motivation or satiation is expected to be temporally correlated. Even 'benign' behavioral variables, such as the location of the animal or the activation of different muscles may change at relatively long time-scales. When recording neural activity related to these variables, any temporal correlations in the neural recording, be it in fMRI, electrophysiology or calcium imaging may result in an erroneous identification of correlates of these behavioral variables because of the temporal correlation confound.

In general, the temporal correlation confound can be addressed by using the permutation analysis of Figure 3, which can provide strong support to the claim that the activity of a particular neuron or voxel co-varies with the behavioral variable. Therefore, the permutation test is a general solution for scientists studying slow processes such as learning. More challenging, however, is precisely identifying what the activity of the neuron represents (for example an action-value or policy). There are no easy solutions to this problem and therefore caution should be applied when interpreting the data.

Another difficulty in identifying action-value neurons is that they are correlated with other decision variables such as policy, state or chosen-value. Therefore, finding a neuron that is significantly correlated with an action-value could be the byproduct of its being modulated by other decision variables, in particular policy. The problem is exacerbated by the fact that standard analyses (e.g., Figure 1D–E) are biased towards classifying neurons as representing action-values at the expense of policy or state.).

As shown in Figure 6, policy representation can be ruled out by finding a representation that is orthogonal to policy, namely state representation. This solution leads us, however, to a serious conceptual issue. All analyses discussed so far are based on significance tests: we divide the space of hypothesis into the ‘scientific claim’ (e.g., neurons represent action-values) and the null hypothesis (e.g., neural activity is independent of the task). An observation that is not consistent with the null hypothesis is taken to support the alternative hypothesis.

The problem we faced with correlated variables is that the null hypothesis and the ‘scientific claim’ were not complementary. A neuron that represents policy is expected to be inconsistent with the null hypothesis that neural activity is independent of the task but it is not an action-value neuron. The solution proposed was to devise a statistical test that seeks to identify a representation that is correlated with action-value and is orthogonal to the policy hypothesis, in order to also rule out a policy representation.

However, this does not rule out other decision-related representations. A ‘pure’ action-value neuron is modulated only by $Q_1$ or by $Q_2$. A ‘pure’ policy neuron is modulated exactly by $Q_1-Q_2$. More generally, we may want to consider the hypotheses that the neuron is modulated by a different combination of the action values, ${a\bullet Q}_1+{b\bullet Q}_2$, where a and b are parameters. For every such set of parameters a and b we can devise a statistical test to reject this hypothesis by considering the direction that is orthogonal to the vector $\left(a,b\right)$. In principle, this procedure should be repeated for every pair of parameters a and b that in not consistent with the action-value hypothesis.

Put differently, in order to find neurons that represent action-values, we first need to define the set of parameters a and b such that a neuron whose activity is modulated by ${a\bullet Q}_1+{b\bullet Q}_2$ will be considered as representing an action-value. Only after this (arbitrary) definition is given, can we construct a set of statistical tests that will rule out the competing hypotheses, namely will rule out all values of a and b that are not in this set. The analysis of Figure 6 implicitly defined the set of a and b such that $a\ne b$ and $a\ne -b$ as the set of parameters that defines action-value representations. In practice, it is already very challenging to identify action-values using the procedure of Figure 6 and going beyond it seems impractical. Therefore, studying the distribution of t-values across the population of neurons may be more useful when studying representations of decision variables than asking questions about the significance of individual neurons.

Importantly, the regression models described in this paper allow us to investigate only some types of representations, namely, linear combinations of the two action-values. However, value representations in learning models may fall outside of this regime. It has been suggested that in decision making, subjects calculate the ratio of action-values (Worthy et al., 2008), or that subjects compute, for each action, the probability that it is associated with the highest value (Morris et al., 2014). Our proposed solution cannot support or refute these alternative hypotheses. If these are taken as additional alternative hypotheses, a neuron should be classified as representing an action-value if its activity is also significantly modulated in the directions that are correlated with action-value and are orthogonal to these hypotheses. Clearly, it is never possible to construct an analysis that can rule out all possible alternatives.

We believe that the confounds that we described have been overlooked because the null hypothesis in the significance tests was not made explicit. As a result, the complementary hypothesis was not explicitly described and the conclusions drawn from rejecting the null hypothesis were too specific. That is, alternative plausible interpretations were ignored. It is important, therefore, to keep the alternative hypotheses explicit when analyzing the data, be it using significance tests or other methods, such as model comparison (Ito and Doya, 2015b).

One may argue that the question of whether neurons represent action-value, policy, state or some other correlated variable is not an interesting question. This is because all these correlated decision variables implicitly encode action-values. Even direct-policy models can be taken to implicitly encode action-values because policy is correlated with the difference between the action-values. However, we believe that the difference between action-value representation and representation of other variables is an important one, because it centers on the question of the computational model that underlies decision making in these tasks. Specifically, the implication of a finding that a population of neurons represents action-values is not that these neurons are involved somehow in decision making. Rather, we interpret this finding as supporting the hypothesis that action-values are explicitly computed in the brain, and that these action-values play a specific role in the decision making process. However, if the results are also consistent with various alternative computational models then this is not the case. Some consider action-value computation to be a necessary part of decision making. By contrast, however, we presented here two models of learning and decision making that do not entail this computation (Figure 2—figure supplement 1, Figure 5). Other examples are discussed in (Mongillo et al., 2014; Shteingart and Loewenstein, 2014) and references therein.

Several trial-design experiments have associated cues with upcoming rewards and reported representations of expected reward, the upcoming action, or the interaction of action and reward (Cromwell and Schultz, 2003; Cromwell et al., 2005; Hassani et al., 2001; Hori et al., 2009; Kawagoe et al., 1998; Pasquereau et al., 2007). Another trial-design experiment reported representation of offer-value and chosen-value in the orbitofrontal cortex (Padoa-Schioppa and Assad, 2006). While such studies do not provide direct evidence for action-value representation, they do provide evidence for representation of closely related decision variables (but see [O'Doherty, 2014]).

The involvement of the basal ganglia in general and the striatum in particular in operant learning, planning and decision-making is well documented (Ding and Gold, 2010; McDonald and White, 1993; O'Doherty et al., 2004; Palminteri et al., 2012; Schultz, 2015; Tai et al., 2012; Thorn et al., 2010; Yarom and Cohen, 2011). However, there are alternatives to the possibility that the firing rate of striatal neurons represents action-values. First, as discussed above, learning and decision making do not entail action-value representation. Second, it is possible that action-value is represented elsewhere in the brain. Finally, it is also possible that the striatum plays an essential role in learning, but that the representation of decision variables there is distributed and neural activity of single neurons could reflect a complex combination of value-related features, rather than ‘pure’ decision variables. Such complex representations are typically found in artificial neural networks (Yamins and DiCarlo, 2016).

Considering the literature, both confounds have been partially acknowledged. Moreover, there have been some attempts to address them. However, as discussed above, even when these confounds were acknowledged and solutions were proposed, these solutions do not prevent the erroneous identification of action-value representation (see Figure 2—figure supplements 4, 5 and 10, Figure 5—figure supplement 1). We therefore conclude that to the best of our knowledge, all studies that have claimed to provide direct evidence that neuronal activity in the striatum is specifically modulated by action-value were either susceptible to the temporal correlations confound (Funamizu et al., 2015; Ito and Doya, 2009; 2015a; Ito and Doya, 2015b; Kim et al., 2013; Kim et al., 2009; Lau and Glimcher, 2008; Samejima et al., 2005; Wang et al., 2013), or reported results in a manner indistinguishable from policy (Cai et al., 2011; FitzGerald et al., 2012; Funamizu et al., 2015; Guitart-Masip et al., 2012; Her et al., 2016; Kim et al., 2013; Kim et al., 2009; Kim et al., 2012; 2007; Stalnaker et al., 2010; Wunderlich et al., 2009). Many studies presented action-value and policy representations separately, but were subject to the second confound (Ito and Doya, 2009; 2015a; Ito and Doya, 2015b; Lau and Glimcher, 2008; Samejima et al., 2005). Furthermore, it should be noted that not all studies investigating the relation between striatal activity and action-value representation have reported positive results. Several studies have reported that striatal activity is more consistent with direct-policy learning than with action-value learning (FitzGerald et al., 2014; Li and Daw, 2011) and one noted that lesions to the dorsal striatum do not impair action-value learning (Vo et al., 2014).

Finally, we would like to emphasize that we do not claim that there is no representation of action-value in the striatum. Rather, our results show that special caution should be applied when relating neural activity to reinforcement-learning related variables. Therefore, the prevailing belief that neurons in the striatum represent action-values must await further tests that address the confounds discussed in this paper.

In order to thoroughly examine the finding of action-value neurons in the striatum, we conducted a literature search to find all the different approaches used to identify action-value representation in the striatum and see whether they are subject to at least one of the two confounds we described here.

The key words ‘action-value’ and ‘striatum’ were searched for in Web-of-Knowledge, Pubmed and Google Scholar, returning 43, 21 and 980 results, respectively. In the first screening stage, we excluded all publications that did not report new experimental results (e.g., reviews and theoretical papers), focused on other brain regions, or did not address value-representation or learning. In the remaining publications, the abstract of the publication was read and the body of the article was searched for ‘action-value’ and ‘striatum’. After this step, articles in which it was possible to find description of action-value representation in the striatum were read thoroughly. The search included PhD theses, but none were found to report new relevant data, not found in papers. We identified 22 papers that directly related neural activity in the striatum to action-values. These papers included reports of single-unit recordings, fMRI experiments and manipulations of striatal activity.

Of these, two papers have used the term action-value to refer to the value of the chosen action (also known as chosen-value) (Day et al., 2011; Seo et al., 2012) and therefore we do not discuss them.

An additional study (Pasquereau et al., 2007) used the expected reward and the chosen action as predictors of the neuronal activity and found neurons that were modulated by the expected reward, the chosen action and their interaction. The authors did not claim that these neurons represent action-values, but it is possible that these neurons were modulated by the values of specific actions. However, the representation of the value of the action when the action is not chosen is a crucial part of action-value representation which differentiates it from the representation of expected reward, and the values of the actions when they were not chosen were not analyzed in this study. Therefore, the results of this study cannot be taken as an indication for action-value representation, rather than other decision variables.

A second group of 11 papers did not distinguish between action-value and policy representations (Cai et al., 2011; Funamizu et al., 2015; Her et al., 2016; Kim et al., 2013; Kim et al., 2009; Wunderlich et al., 2009), or reported policy representation (FitzGerald et al., 2012; Guitart-Masip et al., 2012; Kim et al., 2012; 2007; Stalnaker et al., 2010) in the striatum and therefore their findings do not necessarily imply action-value representation, rather than policy representation in the striatum (see confound 2).

In two additional papers, it was shown that the activation of striatal neurons changes animals’ behavior, and the results were interpreted in the action-value framework (Lee et al., 2015; Tai et al., 2012). However, a change in policy does not entail an action-value representation (see, for example, Figure 5 and Figure 2—figure supplement 1). Therefore, these papers were not taken as strong support to the striatal action-value representation hypothesis.

Finally, six papers correlated action-values, separately from other decision variables, with neuronal activity in the striatum (Ito and Doya, 2009; 2015a; Ito and Doya, 2015b; Lau and Glimcher, 2008; Samejima et al., 2005; Wang et al., 2013). All of them used electrophysiological recordings of single units in the striatum. From these papers, only one utilized an analysis which is not biased towards identifying action-value neurons at the expense of policy and state neurons (Wang et al., 2013). All papers used block-design experiments where action-values are temporally correlated.

Taken together, we concluded that previous reports on action-value representation in the striatum could reflect the representation of other decision variables or temporal correlations in the spike count that are not related to action-value learning.

To model neurons whose firing rate is modulated by an action-value, we considered neurons whose firing rate changes according to:

Where $f\left(t\right)$ is the firing rate in trial $t$, $B=2.5$Hz is the baseline firing rate, $Q_i\left(t\right)$ is the action-value associated with one of the actions $i\in \left\{\mathrm{1,2}\right\}$, $K=2.35$Hz is the maximal modulation and $r$ denotes the neuron-specific level of modulation, drawn from a uniform distribution, $r~U\left[-\mathrm{1,1}\right]$. The spike count in a trial was drawn from a Poisson distribution, assuming a 1 sec-long trial.

To model neurons whose firing rate is modulated by the policy, we considered neurons whose firing rate changes according to:

Where $f\left(t\right)$ is the firing rate in trial $t$, $B=2.5$Hz is the baseline firing rate, $\mathrm{P}\mathrm{r}\left(a\left(t\right)=1\right)$ is the probability of choosing action 1 in trial $t$ that changes in accordance with REINFORCE (Williams, 1992) (see also Figure 5 and corresponding text). $K=3$Hz is the maximal modulation and $r$ denotes the neuron-specific level of modulation, drawn from a uniform distribution, $r~U\left[-\mathrm{1,1}\right]$. The spike count in a trial was drawn from a Poisson distribution, assuming a 1 sec-long trial.

In the covariance based plasticity model the decision-making network is composed of two populations of Poisson neurons: each neuron is characterized by its firing rate and the spike count of a neuron in a trial (1 sec) is randomly drawn from a Poisson distribution. The chosen action corresponds to the population that fires more spikes in a trial (Loewenstein, 2010; Loewenstein and Seung, 2006). At the end of the trial, the firing rate of each of the neurons (in the two population) is updated according to $f\left(t+1\right)=f\left(t\right)+\eta \bullet R\left(t\right)\bullet \left(s\left(t\right)-f\left(t\right)\right)$, where $f\left(t\right)$ is the firing rate in trial $t$, $\eta =0.07$ is the learning rate, $R\left(t\right)$ is the reward delivered in trial $t$ ($R\left(t\right)\in \left\{\mathrm{0,1}\right\}$ in our simulations) and $s\left(t\right)$ is the measured (realized) firing rate in that trial, that is the spike count in the trial. The initial firing rate of all simulated neurons is 2.5Hz. The network model was tested in the operant learning task of Figure 1. A session was terminated (without further analysis) if the model was not able to choose the better option more than 14 out of 20 consecutive times for at least 200 trials in the same block. This occurred on 20% of the sessions. We simulated two populations of 1,000 neurons in 500 successful sessions. Note that because on average, the empirical firing rate is equal to the true firing rate, $f\left(t\right)=⟨s\left(t\right)⟩$, changes in the firing rate are driven, on average, by the covariance of reward and the empirical firing rate: $⟨∆f\left(t\right)⟩\equiv ⟨f\left(t+1\right)-f\left(t\right)⟩=\eta \bullet \mathrm{c}\mathrm{o}\mathrm{v}\left(R\left(t\right),s\left(t\right)\right)$(Loewenstein and Seung, 2006). The estimated action-values in Figure 2—figure supplement 1 were computed from the actions and rewards of the covariance model by assuming the Q-learning model (Equations 1 and 2).

The data in Figure 2—figure supplement 2A–B was recorded by Oren Peles in Eilon Vaadia's lab. It was recorded from one female monkey (Macaca fascicularis) at 3 years of age, using a 10 × 10 microelectrode array (Blackrock Microsystems) with 0.4 mm inter-electrode distance. The array was implanted in the arm area of M1, under anesthesia and aseptic conditions.

Behavioral Task: The Monkey sat in a behavioral setup, awake and performing a Brain Machine Interface (BMI) and sensorimotor combined task. Spikes and Local Field Potentials were extracted from the raw signals of 96 electrodes. The BMI was provided through real time communication between the data acquisition system and a custom-made software, which obtained the neural data, analyzed it and provided the monkey with the desired visual and auditory feedback, as well as the food reward. Each trial began with a visual cue, instructing the monkey to make a small hand movement to express alertness. The monkey was conditioned to enhance the power of beta band frequencies (20-30Hz) extracted from the LFP signal of 2 electrodes, receiving a visual feedback from the BMI algorithm. When a required threshold was reached, the monkey received one of 2 visual cues and following a delay period, had to report which of the cues it saw by pressing one of two buttons. Food reward and auditory feedback were delivered based on correctness of report. The duration of a trial was on average 14.2s. The inter-trial-interval was 3s following a correct trial and 5s after error trials. The data used in this paper, consists of spiking activity of 89 neurons recorded during the last second of inter-trial-intervals, taken from 600 consecutive trials in one recording session. Pairwise correlations were comparable to previously reported (Cohen and Kohn, 2011), $r_{SC}=0.047\pm 0.17$ (SD), ($r_{SC}=0.037\pm 0.21$ for pairs of neurons recorded from the same electrode).

Animal care and surgical procedures complied with the National Institutes of Health Guide for the Care and Use of Laboratory Animals and with guidelines defined by the Institutional Committee for Animal Care and Use at the Hebrew University.

The auditory cortex recordings appearing in Figure 2—figure supplement 2C–D are described in detail in (Hershenhoren et al., 2014). In short, membrane potential was recorded intracellularly in the auditory cortex of halothane-anesthetized rats. The data consists of 125 experimental sessions recorded from 39 neurons. Each session consisted of 370 pure tone bursts. Tone duration was 50 ms with 5 ms linear rise/fall ramps. In the data presented here, trials began 50 ms prior to the onset of the tone burst. For each session, all trials were either 300 msec or 500 msec long. Trial length remained identical throughout a session and depended on smallest interval between two tones in each session. Spike events were identified following high pass filtering with a corner frequency of 30Hz. Local maxima that were larger than 60 times the median of the absolute deviation from the median (MAD) were classified as spikes. The data presented here consists only of the spike counts in each trial, rather than the full membrane potential trace.

The basal ganglia recordings that are analyzed in Figure 3 and Figure 2—figure supplement 3 are described in detail in (Ito and Doya, 2009). In short, rats performed a combination of a tone discrimination task and a reward-based free-choice task. Extracellular voltage was recorded in the behaving rats from the NAc and VP using an electrode bundle. Spike sorting was done using principal component analysis. In total, 148 NAc and 66 VP neurons across 52 sessions were used for analyses (In 18 of the 70 behavioral sessions there were no neural recordings).

To imitate experimental procedures, we regressed the spike counts on estimates of the action-values, rather than the subjective action-values that underlay model behavior (to which the experimentalist has no direct access). For that goal, for each session, we assumed that $Q_i\left(1\right)=0.5$ and found the set of parameters $\widehat{\alpha }$ and $\widehat{\beta }$ that yielded the estimated action-values that best fit the sequences of actions in each experiment by maximizing the likelihood of the sequence. Action-values were estimated from Equation 1, using these estimated parameters and the sequence of actions and rewards. Overall, the estimated values of the parameters $\alpha$ and $\beta$ were comparable to the actual values used: on average, $\widehat{\alpha }=0.12\mathrm{}\pm 0.09$ (standard deviation) and $\widehat{\beta }=2.6\mathrm{}\pm 0.7$ (compare with $\alpha$=0.1 and $\beta$=2.5).

Following standard procedures (Samejima et al., 2005), a sequence of spike-counts, either simulated or experimentally measured was excluded due to low firing rate if the mean spike count in all blocks was smaller than 1. This procedure excluded 0.02% (4/20,000) of the random-walk neurons and 0.03% (285/1,000,000) of the covariance-based plasticity neurons. Considering the auditory cortex recordings, we assigned each of the 125 spike counts to 40 randomly-selected sessions. 23% of the neural recordings (29/125) were excluded in all 40 sessions. Because blocks are defined differently in different sessions, some neural recordings were excluded only when assigned to some sessions but not others. Of the remaining 96 recordings, 14% of the recordings × sessions were also excluded. Similarly, considering the basal ganglia neurons, we assigned each of the 642 recordings (214 × 3 phases) to 40 randomly-selected sessions. 11% (74/(214 × 3)) of the recordings were excluded in all 40 sessions. Of the remaining 568 recordings, 9% of the recordings × sessions were also excluded. None of the simulated action-value neurons (0/20,000) or the motor cortex neurons (0/89) were excluded.

The computation of the t-values of the regression of the spike counts on the estimated action-values (as in Figures 1, 2 and 5, Figure 2—figure supplement 1, – Figure 2—figure supplement 2, –Figure 2—figure supplement 3) was done using the following regression model:

Where $\mathrm{s}\left(\mathrm{t}\right)$ is the spike count in trial $t$, $Q_1\left(t\right)$ and $Q_2\left(t\right)$ are the estimated action-values in trial $t$, $ϵ\left(t\right)$ is the residual error in trial $t$ and ${\beta }_{0-2}$ are the regression parameters.

The computation of the t-values of the regression of the spike counts on the reward probabilities in the trial design experiment (as in Figure 4) was done using the following regression model:

Where ${\displaystyle t}$ denotes the trial. Only the last 200 trials of the session were anlyzed. $s\left(t\right)$ is the mean spike count, ${RP}_1\left(t\right)$ and ${RP}_2\left(t\right)$ are the reward probabilities corresponding to action 1 or action 2, respectively (in this experimental design $RP$ could be 0.1,0.5 or 0.9), $ϵ\left(t\right)$ is the residual error and ${\beta }_{0-2}$ are the regression parameters.

The computation of the t-values of the regression of the spike counts on state and policy in a trial design experiment (as in Figure 6) was done using the following regression model:

All variables and parameters are the same as in Equation 7

All regression analyses were done using regstats in MATLAB (version 2016A).

To compare the spike counts of the example neurons, in the last 20 trials of each block (Figure 1B; Figure 2—figure supplement 1B; Figure 2—figure supplement 2A; Figure 2—figure supplement 2C; Figure 2A) we executed the Wilcoxon rank sum test, using ranksum in MATLAB. All tests were two-tailed.

Significance of t-values slightly depends on session length. For the session lengths we considered, 0.05 significance bounds varied between 1.962 and 1.991. For consistency, we chose a single conservative bound of 2. Similarly, 0.025 and 0.01 significance bounds were chosen to be 2.3 and 2.64, respectively.

For all significance boundaries the false positive thresholds were computed naively, that is, assuming the analysis is not confounded in any way and that the two predictors are not correlated with each other. For example, assuming the false positive rate from a single t-test for a significant regression coefficient is $P$, for the standard analysis, the false positive rate for each action-value classification was defined as $P\bullet (1-P)$, and the false positive rate was equal for state and policy classification and was defined as $P^2/2$. In Figure 6 the false positive rate computed for random-walk neurons was $P^2/2$ for each action-value classification, and the false positive rate computed for state or policy neurons was $P/2$ for each action-value classification.

For each action-value and random-walk neuron, we computed the t-values of the regressions of its spike-count on estimated action-values from the sessions of Figure 1E. Because the number of trials can affect the distribution of t-values, we only considered in our analysis the first 170 trials of the 504 sessions longer or equal to 170 trials. This number, which is approximately the median of the distribution of number of trials per session, was chosen as a compromise between the number of trials per session and number of sessions. When performing the permutation test on the basal ganglia data we included all recordings and only the first 332 trials in each session, which is the smallest number of trials used in a session in this dataset.

Two points are noteworthy. First, the distribution of the t-values of the regression of the spike count of a neuron on all action-values depends on the neuron (see difference between distributions in Figure 3A). Similarly, the distribution of the t-values of the regression of the spike counts of all neurons on an action-value depends on the action-value (not shown). Therefore, the analysis could be biased in favor (or against) finding action-value neurons if the number of neurons analyzed from each session (and therefore are associated with the same action-values) differs between sessions. Second, this analysis does not address the correlated decision variables confound.

Finally, we would like to point out that there is an alternative way of performing the permutation test, which is applicable when the number of sessions is small, while the number of neurons recorded in a session is large. Instead of comparing the t-values from the regression of a neuron on different action-values, one can compare the t-values from different neurons on the same action-value. However, this method is only applicable under the assumption that the temporal correlations that are not related to action-value in the neuronal activity are similar between sessions.

In Figure 2—figure supplement 4 we considered the experiment and analysis described in (Kim et al., 2009). That experiment consisted of four blocks, each associated with a different pair of reward probabilities, (0.72, 0.12), (0.12, 0.72), (0.21, 0.63) and (0.63, 0.21), appearing in a random order, with the better option changing location with each block change. The number of trials in a block was preset, ranging between 35 and 45 with a mean of 40 (this is unlike the experiment described in Figure 1, in which termination of a block depended on performance).

First, we used Equations 1 and 2 to model learning behavior in this protocol. Then, we estimated the action-values according to choice and reward sequences, as in Figure 1. These estimated action-values were used for regression of the spike counts of the random-walk, motor cortex, auditory cortex, and basal ganglia neurons in the following way: each spike count sequence was randomly assigned to a particular pair of estimated action-values from one session. The spike count sequence was regressed on these estimated action-values. The resultant t-values were compared with the t-values of 1000 regressions of the spike-count, permuted within each block, on the same action-values. The p-value of this analysis was computed as the percentage of t-values from the permuted spike-counts that were higher in absolute value than the t-value from the regression of the original spike count. The significance boundary was set at p<0.025 (Kim et al., 2009). Neurons with at least one significant regression coefficient (rather than exactly one significant regression coefficient) were classified as action-value modulated neurons (Kim et al., 2009).

Following (Asaad et al., 2000) we conducted an additional analysis with repeating blocks. We simulated learning behavior in the same experiment as in Figure 2—figure supplement 10. This experiment is composed of 8 blocks - the 4 blocks of Figure 1, repeated twice, in random permutation. We restricted our analysis to the 438 sessions with 332 trials or fewer (332 trials is the shortest session in the basal ganglia recording). Each spike count was analyzed 40 times, using 40 randomly-assigned sessions. For each block, we restricted the analysis to the neuronal activity in the last 20 trials of the block.

First, we conducted four one-way ANOVAs (using MATLAB’s anova1) to compare the neuronal activities in blocks associated with the same action-values (e.g., the neuronal activity in the two blocks, in which reward probabilities were (0.1,0.5)). Neurons were excluded from further analysis if we found a significant difference in their firing rates in at least one of these comparisons (df(columns)=1, df(error)=38, p<0.1). This procedure excludes from further analysis ‘drifting’ neurons, whose spike count significantly varied in the session.

Next, for each action-value we conducted a one-way ANOVA (using MATLAB’s anova1), which compared the neuronal activity between the two blocks in which the action-value was 0.1 and the two blocks in which the action-value was 0.9 (df(columns)=1, df(error)=78, p<0.01). We classified neurons as representing action-values if there was a significant difference between their firing rates for one action-value but not for the other.

Despite the removal of ‘drifting’ neurons, this analysis yielded an erroneous classification of action-value neurons in all datasets: random-walk neurons, 18%; motor cortex neurons, 12%; auditory cortex neurons, 5%; basal ganglia neurons, 9%. This is despite the fact that the expected false positive rate is only 2%. These results indicate that the exclusion of ‘drifting’ neurons as in (Asaad et al., 2000) does not solve the temporal correlations confound.

Data from the motor cortex, auditory cortex, and basal ganglia was the same as in Figure 2—figure supplements 2–3. Data for random-walk included 1000 newly simulated neurons, using the same parameters as in Figure 2 (this was done to create enough trials in each simulated spike count).

The data of the basal ganglia recordings from (Ito and Doya, 2009) is available online at https://groups.oist.jp/ncu/data and was analyzed with permission from the authors. Motor cortex data (recorded by Oren Peles in Eilon Vaadia's lab) and auditory cortex data (taken from the recordings in (Hershenhoren et al., 2014)) is available at https://github.com/lotem-elber/striatal-action-value-neurons-reconsidered-codes (Elber-Dorozko and Loewenstein, 2018). The custom MATLAB scripts used to create simulated neurons and to analyze simulated and recorded neurons are also available at https://github.com/lotem-elber/striatal-action-value-neurons-reconsidered-codes (Elber-Dorozko and Loewenstein, 2018; copy archived at https://github.com/elifesciences-publications/striatal-action-value-neurons-reconsidered-codes).

• Yonatan Loewenstein

• Yonatan Loewenstein

• Yonatan Loewenstein