Direct observation of the neural computations underlying a single decision
eLife assessment
This fundamental work quantifies the stochastic dynamics of neural population activity in the lateral intraparietal area (LIP) of the macaque monkey brain during single perceptual decisions. These singletrial dynamics have been subject to intense debate in neuroscience, and they have significant implications for modeling decisionmaking in various fields including neuroscience and psychology. Through a combination of stateoftheart recordings from many LIP neurons and theorydriven data analyses, the authors provide convincing evidence for the notion that singletrial neural population dynamics in LIP encode the decision variable postulated by the driftdiffusion model of decisionmaking.
https://doi.org/10.7554/eLife.90859.3.sa0Fundamental: Findings that substantially advance our understanding of major research questions
 Landmark
 Fundamental
 Important
 Valuable
 Useful
Convincing: Appropriate and validated methodology in line with current stateoftheart
 Exceptional
 Compelling
 Convincing
 Solid
 Incomplete
 Inadequate
During the peerreview process the editor and reviewers write an eLife Assessment that summarises the significance of the findings reported in the article (on a scale ranging from landmark to useful) and the strength of the evidence (on a scale ranging from exceptional to inadequate). Learn more about eLife Assessments
Abstract
Neurobiological investigations of perceptual decisionmaking have furnished the first glimpse of a flexible cognitive process at the level of single neurons. Neurons in the parietal and prefrontal cortex are thought to represent the accumulation of noisy evidence, acquired over time, leading to a decision. Neural recordings averaged over many decisions have provided support for the deterministic rise in activity to a termination bound. Critically, it is the unobserved stochastic component that is thought to confer variability in both choice and decision time. Here, we elucidate this driftdiffusion signal on individual decisions. We recorded simultaneously from hundreds of neurons in the lateral intraparietal cortex of monkeys while they made decisions about the direction of random dot motion. We show that a single scalar quantity, derived from the weighted sum of the population activity, represents a combination of deterministic drift and stochastic diffusion. Moreover, we provide direct support for the hypothesis that this driftdiffusion signal approximates the quantity responsible for the variability in choice and reaction times. The populationderived signals rely on a small subset of neurons with response fields that overlap the choice targets. These neurons represent the integral of noisy evidence. Another subset of directionselective neurons with response fields that overlap the motion stimulus appear to represent the integrand. This parsimonious architecture would escape detection by statespace analyses, absent a clear hypothesis.
Introduction
Neural signals in the mammalian cortex are notoriously noisy. They manifest as a sequence of action potentials (spikes) that approximate nonstationary Poisson point processes. Therefore, to characterize the signal produced by a neuron, electrophysiologists typically combine the spike times from many repetitions or trials relative to the time of an event (e.g., stimulus onset) to yield the average firing rate of the neuron as a function of time. Such trialaveraged firing rates are the main staple of systems neuroscience (Here and throughout, trial average and acrosstrial average refer to the mean of signal values, over all specified trials at the same time, $t$, relative to a trial event (e.g., motion onset)). They are the source of knowledge about spatial selectivity (e.g., receptive fields), feature selectivity (e.g., direction of motion, faces vs. other objects), and even cognitive signals associated with working memory, anticipation, attention, motor planning, and decisionmaking. But there is an important limitation.
Trial averages suppress signals that vary independently across trials. In many cognitive tasks, such as difficult decisions, the variable component of the signal is the most interesting because it is this component that is thought to explain the variable choice and response time. This variability is thought to arise from a decision process that accumulates noisy evidence in favor of the alternatives and terminates when the accumulated evidence for one alternative, termed the decision variable (DV), reaches a terminating bound. The DV is stochastic because the integral of noisy samples of evidence is biased Brownian motion (or driftdiffusion) and this leads to a stochastic choice and response time on each decision. However, the stochastic part of this signal is suppressed by averaging across trials. We will use the term driftdiffusion because it is the expression most commonly applied in models of decisionmaking (Ratcliff and Rouder, 1998; Gold and Shadlen, 2007), and we will consider the noise part—that is, diffusion—as the signal of interest.
In the setting of difficult perceptual decisions, studied here, bounded driftdiffusion reconciles the relationship between decision speed and accuracy. It also explains the trialaveraged firing rates of neurons in the lateral intraparietal area (LIP) that represent the action used by monkeys to indicate their choice. These firing rate averages show motiondependent, ramping activity that reflects the direction and strength of motion, consistent with the drift component of driftdiffusion (Roitman and Shadlen, 2002). Up to now, however, the diffusion component has not been observed, owing to averaging.
There is thus a missing link between the mathematical characterization of the decision process and its realization in neural circuits, leaving open the possibility that driftdiffusion dynamics do not underlie LIP activity (e.g., Latimer et al., 2015), or emerge only at the level of the population, without explicit representation by single neurons. We reasoned that these and other alternatives to driftdiffusion could be adjudicated if it were possible to resolve the DV giving rise to a single decision.
This stratagem is now feasible, owing to the development of highdensity Neuropixels probes, which are capable of recording from deep sulci in the primate brain. Here we provide the first direct evidence for a driftdiffusion process underlying single decisions. We recorded simultaneously from up to 203 neurons in area LIP while monkeys made perceptual decisions about the direction of dynamic random dot motion (Newsome et al., 1989; Gold and Shadlen, 2007). Using a variety of dimensionality reduction techniques, we show that a driftdiffusion signal can be detected in such populations on individual trials. Moreover, this signal satisfies the criteria for a DV that controls the choice and reaction time (RT). Notably, the signal of interest is dominated by a small subpopulation of neurons with response fields that overlap one of the choice targets, consistent with earlier singleneuron studies (e.g., Shadlen and Newsome, 1996; Roitman and Shadlen, 2002; Churchland et al., 2011; Gold and Shadlen, 2007).
Results
Two monkeys made perceptual decisions, reported by an eye movement, about the net direction of dynamic random dot motion (Figure 1a). We measured the speed and accuracy of these decisions as a function of motion strength (Figure 1b, circles). The choice probabilities and the distribution of RTs are well described (Figure 1b, traces) by a bounded driftdiffusion model (Figure 1c). On 50% of the trials, a brief (100 ms) pulse of weak leftward or rightward motion was presented at a random time. The influence of these pulses on choice and RT further supports the assertion that the choices and RTs arose through a process of integration of noisy samples of evidence to a stopping bound (Figure 1—figure supplement 1; Stine et al., 2020; Stine et al., 2023; Hyafil et al., 2023).
In addition to the main task, the monkeys performed two control tasks: instructed, delayed saccades to peripheral targets and passive viewing of random dot motion (see ‘Methods’). These control tasks served to identify, post hoc, neurons with response fields that overlap the choice target in the hemifield contralateral to the recording site (${\text{T}}_{\text{in}}^{\text{con}}$), neurons with response fields that overlap the other choice target (${\text{T}}_{\text{in}}^{\text{ips}}$), and neurons with response fields that overlap the randomdot motion stimulus (${\text{M}}_{\text{in}}$; Table 1).
We recorded simultaneously from populations of neurons in area LIP using newly developed macaque Neuropixels probes (Trautmann et al., 2023) while monkeys performed these tasks. The data set comprises eight sessions from two monkeys (1696–2894 trials per session; Table 2). Our primary goal was to identify activity in LIP that relates to the DV, a theoretical quantity that determines the choice and RT on each trial. To achieve this, we formed weighted averages from all neurons in the sample population, including those with response fields that do not overlap a choice target or the motion stimulus. We used several strategies to assign this vector of weights, which we refer to as a coding direction in the neuronal state space (NSS). The projection of the spiking activity from the population of neurons onto the vector of weights gives rise to a scalar function of time, ${S}^{x}(t)$, where the superscript $x$ labels the strategy. We focus on such onedimensional projections because of the longstanding hypothesis that the DV is driftdiffusion, which is a scalar function of time.
We first developed a targeted strategy that would reproduce the wellknown coherencedependent ramping activity evident in the acrosstrial averages. This strategy applies regression to best approximate a linear ramp, on each trial, $i$, that terminates with a saccade to the choice target contralateral to the hemisphere of the LIP recordings. The ramps are defined on the epoch spanning the decision time: from ${t}_{0}=0.2$ s after motion onset to ${t}_{1}=0.05$ s before saccade initiation (black lines in Figure 2—figure supplement 1) The epoch is motivated by many previous studies (see Gold and Shadlen, 2007; Shadlen and Kiani, 2013, for reviews). Each ramp begins at ${f}_{i}({t}_{0})=1$ and ends at ${f}_{i}({t}_{1})=1$. The ramp approximates the expectation—conditional on the choice and response time—of the deterministic components of the driftdiffusion signal, which, in addition to the drift, can incorporate (i) a timedependent but evidenceindependent urgency signal (Churchland et al., 2008; Drugowitsch et al., 2012), and (ii) a dynamic bias signal (Hanks et al., 2011). It can also be viewed as an approximation to firing rates averaged across trials and grouped by contraversive choice and RT quantile (Figure 2—figure supplement 2). Importantly, the fit is not guided by an assumption of an underlying diffusion process. That is, the ramp coding direction is agnostic to the underlying processes whose averages approximate ramps. The weights derived from these regression fits specify a ramp coding direction in the state space defined by the population of neurons in the session. The singletrial signal, ${S}^{\mathrm{ramp}}(t)$, is rendered as the projection of the population firing rates onto this coding direction.
The left side of Figure 2a shows singletrial activity rendered by this strategy. The right side of the figure shows the averages of the singletrial responses grouped by signed coherence and aligned to motion onset or response time (saccade initiation). These averaged traces exhibit features of the firing rate averages in previous studies of single neurons in LIP (e.g., Roitman and Shadlen, 2002). They begin to diverge as a function of the direction and strength of motion approximately 200 ms after the onset of motion. The traces converge near the time of saccadic response to the contralateral choice target such that the coherence dependence is absent or greatly diminished. Coherence dependence remains evident through the initiation of saccades to the right (ipsilateral) target, consistent with a race architecture—between negatively correlated accumulators—depicted in Figure 1c.
We complemented this regression strategy with principal component analysis (PCA) and use the first PC (PC1), which explains $44\pm 7$% of the variance (mean ± s.e. across sessions) of the activity between 200 and 600 ms from motion onset (see ‘Methods’). This coding direction renders singletrial signals, ${S}^{\text{PC1}}(t)$ (Figure 2b). In a third strategy, we consider the mean activity of neurons with response fields that overlapped the contralateral choice target (${\text{T}}_{\text{in}}^{\text{con}}$ neurons), which were the focus of previous singleneuron studies (e.g., Shadlen and Newsome, 1996; Platt and Glimcher, 1999; Roitman and Shadlen, 2002). In those studies, the task was modified so that one of the choice targets was placed in the neural response field, whereas here we identify neurons post hoc with response fields that happen to overlap the contralateral choice target. This difference probably accounts for the lower firing rates of the ${\text{T}}_{\text{in}}^{\text{con}}$ neurons studied here. Figure 2c shows singletrial and acrosstrial averages from these ${\text{T}}_{\text{in}}^{\text{con}}$ neurons. They too render signals, ${S}_{\text{Tin}}^{\text{con}}(t)$, similar to those derived from the full population. The ${\text{T}}_{\text{in}}^{\text{con}}$ neurons thus furnish a third coding direction defined by a vector of identical positive weights assigned to all ${\text{T}}_{\text{in}}^{\text{con}}$ neurons and 0 for all other neurons in the population. The emboldened singletrial traces in Figure 2 (left) correspond to the same trials rendered by the three coding directions. It is not difficult to tell which are the corresponding traces, an observation that speaks to their similarity, and the same is true for the averages. We will expand on this observation in what follows.
The averages show the deterministic drift component of the hypothesized driftdiffusion process, with the slope varying monotonically with the signed motion strength (Figure 2, right). The rise begins to saturate as a consequence of the putative termination bound—a combination of dropout of trials that are about to terminate and the effect on the distribution of possible diffusion paths imposed by the very existence of a stopping bound. This saturation is evident earlier on trials with stronger motion, hence shorter RT, on average. The positive buildup rate on the 0% coherence motion represents the timedependent, evidenceindependent signal that is thought to reflect the cost of time. It leads to termination even if the evidence is weak, equivalent to collapsing stopping bounds in traditional, symmetric driftdiffusion models (Drugowitsch et al., 2012). Removal of this urgency signal, $u(t)$, from the nonzero coherence traces renders the positive and negative coherence averages symmetric relative to zero on the ordinate (Figure 2—figure supplement 3).
The singletrial responses in Figure 2 do not look like the averages but instead approximate driftdiffusion. We focus on the epoch from 200 to 500 (or 600) ms from motion onset—that is, the first 300 (or 400) ms of the period in which the averages reflect the integration of evidence. Some traces are cut off before the end of the epoch because a saccade occurred 100 ms later on the trial. However, most 0% coherence trials continue beyond 500 ms (median RT $>600$ ms). The singletrial traces do not rise monotonically as a function of time but meander and tend to spread apart from each other vertically. For unbounded diffusion, the variance would increase linearly, but as just mentioned, the existence of an upper stopping bound and the limited range of firing rates (e.g., nonnegative) renders the function sublinear at later times (Figure 3a, Figure 3—figure supplement 1). The autocorrelation between an early and a later sample from the same diffusion trace is also clearly specified for unbounded diffusion. The theoretical values shown in Figure 3b and c are the autocorrelations of unbounded diffusion processes that are smoothed identically to the neural signals (see ‘Methods’ and Appendix 1). The autocorrelations in the data follow a strikingly similar pattern. These observations support the assertion that the coherencedependent (ramplike) firing rate averages observed in previous studies of area LIP are composed of stochastic driftdiffusion processes on single trials.
Singletrial driftdiffusion signals approximate the decision variable
We next evaluate the hypothesis that the driftdiffusion signal, ${S}^{x}(t)$, is the DV that controls the choice and response time. We have identified several coding directions that produce candidate DVs, and as we will see below, there are also other coding directions of interest that can be derived from the population. Additionally, PCA indicates that the dimensionality of the data is low, but greater than 1 (participation ratio = 4.4 ± 1.3; Mazzucato et al., 2016; Gao et al., 2017). Therefore, one might wonder whether it is sensible to assume that the DV can be approximated by a scalar measure arising from a single coding direction as opposed to a higher dimensional representation. Two decoding exercises are adduced to support the assumption.
We constructed a logistic decoder of choice using each neuron’s spike counts in 50 ms bins between 100 and 500 ms after motion onset. As shown in Figure 4a, this Whatdecoder (orange) predicts choice as accurately as a decoder of simulated data from a driftdiffusion model (black) using parameters derived from fits to the monkeys’ choice and RT data (see ‘Methods’). The simulation establishes a rough estimate of the decoding accuracy that can be achieved, given the stochastic nature of the choice, were we granted access to the driftdiffusion signal that actually determines the decision. In this analysis, the decoder can use a different vector of weights at each point in time (timedependent coding directions; see Peixoto et al., 2021). However, if the representation of the DV in LIP is onedimensional, then a decoder trained at one time should perform well when tested at a different time. The red curve in Figure 4a shows the performance of a Whatdecoder with a fixed trainingtime (450 ms after motion onset; red arrow). This decoder performs nearly as well as the decoder trained at each time bin. The heatmap (Figure 4b) generalizes this observation. It shows two main features for all times $300<t<500$ ms (dashed box). First, unsurprisingly, for a What choice decoder trained on data at one time $t=x$, the predictions improve as the testing time advances (the decoding accuracy increases along any vertical) as more evidence is accrued. Second, and more importantly, decoders tested at time $t=y$ perform similarly, independent of when they were trained (there is little variation in decoding accuracy along any horizontal). This observation suggests that a single vector of weights may suffice to decode the choice from the population response.
The second decoder is trained to predict whether a saccade to the contralateral choice target will be initiated in the next 150 ms. This Whendecoder is trained by logistic regression to predict a binary output: 1 at all timepoints that are within 150 ms of an upcoming saccade and 0 elsewhere (Figure 4—figure supplement 1). We validated the Whendecoder by computing the area under an ROC (AUC) using the heldout (odd) trials (mean AUC over all time points: 0.84), but this is tangential to our goal. Although the Whendecoder was trained only to predict the time of saccades, our rationale for developing this decoder was to test whether the When coding direction can be used to predict the choice. The green trace in Figure 4a shows the accuracy of ${S}^{\text{When}}(t)$ to predict the choice. The performance is almost identical to the choice decoder, despite being trained on a temporal feature of trials ending in the same left choice. This feat is explained by the similarity of signals produced by the When and other coding directions. Note the similarity of the trialaveraged ${S}^{\text{When}}$ signals displayed in Figure 4c to those in Figure 2 (see also Figure 4—figure supplement 2, right). Indeed, the cosine similarity between the When and Ramp coding directions is $0.67\pm 0.03$ (Figure 4d). In light of this, it is not surprising that the weighting vectors derived from both the What and Whendecoders also render singletrial driftdiffusion traces that resemble each other and those rendered by other coding directions (Figure 4e). Together these analyses support the assertion that the DV is likely to be captured by a single dimension, consistent with Ganguli et al., 2008.
If the onedimensional signals, ${S}^{x}(t)$, approximate the DV, they should explain the variability of choice and RT for trials sharing the same direction and motion strength. Specifically, (i) early samples of ${S}^{x}(t)$ should be predictive of choice and correlate inversely with the RT on trials that result in contraversive (leftward) choices, (ii) later samples ought to predict choice better and correlate more strongly (negatively) with RT than earlier samples, and (iii) later samples should contain the information present in the earlier samples and thus mediate (i.e., reduce the leverage) of the earlier samples on choice and RT. Each of these predictions is borne out by the data.
The analyses depicted in Figure 5 allow us to visualize the influence of the singletrial signals, ${S}^{x}(t)$, on the choice and RT on that trial. We focus on the early epoch of evidence accumulation (200–550 ms after random dot motion onset) and restrict the analyses to decisions with $\text{RT}\ge 670$ ms and $\text{coherence}\le 6.4\%$. The RT restriction eliminates 17% of the eligible trials. Larger values of ${S}^{x}(t)$ are associated with a larger probability of a left (contraversive) choice and a shorter RT, hence negative correlation between ${S}^{x}(t)$ and RT. We use the term, leverage, to describe the strength of both of these associations. The leverage on choice (Figure 5a, black traces) is the contribution of ${S}^{x}(t)$ to the log odds of a left choice, after accounting for the motion strength and direction (i.e., the coefficients, ${\beta}_{1}(t)$ in Equation 8). The leverage on RT (Figure 5b) is the Pearson correlation between ${S}^{x}(t)$ and the RT on that trial, after accounting for the effect of motion strength and direction on ${S}^{x}$ and RT (see ‘Methods’). The leverage is evident from the earliest sign of evidence accumulation, 200 ms after motion onset, and its magnitude increases as a function of time, as evidence accrues (Figure 5, top). The filled circle to the right of the traces in each graph shows the leverage of ${S}^{x}$ at $t=550$ ms, which is 120 ms before any of the included trials have terminated. Both observations are consistent with the hypothesis that ${S}^{x}$ represents the integral of noisy evidence used to form and terminate the decision. Two control analyses demonstrate that the degree of leverage on choice and RT do not arise by chance: (i) random coding directions in state space produce negligible leverage (Figure 5—figure supplement 1, top), and (ii) breaking the trialbytrial correspondence between neural activity and behavior eliminates all leverage (see Reviewer Figure 1 in reply to peer review).
Importantly, the leverage at earlier times is mediated by the later sample at $t=550$ ms. The blue traces in all graphs show the remaining leverage once this later sample is allowed to explain the choice and RT—by including, respectively, an additional term in the logistic regression (Equation 9) and calculating the partial correlation, conditional on ${S}^{x}(t=0.55)$. We assessed statistical significance of the mediation statistics, ${\xi}^{\text{Ch}}$ and ${\xi}^{\text{RT}}$ (Equations 7 and 10) in each session for the three signals shown in Figure 5 using a bootstrap procedure (see ‘Methods’, Equation 10). Mediation is significant in 47 of the 48 comparisons (all p<0.023, median $\mathrm{p}<{10}^{317}$). The one nonsignificant comparison is ${\xi}^{\text{Ch}}$ for ${S}_{\text{Tin}}^{\text{con}}$ in session 2 (p=0.73). The mediation is significant when this comparison is included in the combined data ($\mathrm{p}<{10}^{9}$, paired samples ttest). The stark decrease in leverage is consistent with onedimensional diffusion in which later values of the signal contain the information in the earlier samples plus what has accrued in the interim. Had we recorded from all the neurons that represent the DV, we would expect the mediation to be complete (e.g., partial correlation = 0). However, our recorded population is only a fraction of the entire population. Indeed, the observed degree of mediation is similar to values obtained from simulations of weakly correlated, noisy neurons (Figure 5—figure supplement 1, bottom).
There is one additional noteworthy observation in Figure 5 that highlights the importance of the ${\text{T}}_{\text{in}}^{\text{con}}$ neurons. The top and middle rows (${S}^{\mathrm{ramp}}$ and ${S}^{\text{PC1}}$) contain a second, open symbol, which is simply a copy of the filled symbol from the bottom row (${S}_{\text{Tin}}^{\text{con}}$). The yellow traces show significant crossmediation of ${S}^{\mathrm{ramp}}$ and ${S}^{\text{PC1}}$ by the sample, ${S}_{\text{Tin}}^{\text{con}}(t=0.55)$ (all $\mathrm{p}<0.05$; median $\mathrm{p}<{10}^{268}$; bootstrap as above). This signal, carried by 9–21% of the neurons, mediates signals produced by the full population of 54–203 neurons nearly as strongly as ${S}^{\mathrm{ramp}}$ and ${S}^{\text{PC1}}$ mediate themselves. The observation suggests that minimal leverage is gained by sophisticated analyses of the full NSS compared to a simple average of ${\text{T}}_{\text{in}}^{\text{con}}$ neurons. This is both reassuring and disquieting: reassuring because the ${\text{T}}_{\text{in}}^{\text{con}}$ neurons compose the dominant projection from LIP to the portions of the superior colliculus (SC) and the frontal eye field involved in the generation of saccades toward the contralateral choice target (Paré and Wurtz, 1997; Ferraina et al., 2002); disquieting because the functional relevance of these neurons is not revealed by the other coding directions. The weights assigned to the ${\text{T}}_{\text{in}}^{\text{con}}$ neurons span all percentiles (mean IQR: 49–96; mean 71st percentile, $AUC=0.74\pm 0.05$) in the ramp coding direction. They contribute disproportionately to PC1 and the What and When decoders but not enough to stand out based on their weights. Indeed, the ability to predict that a neuron is ${\text{T}}_{\text{in}}^{\text{con}}$ from its weight or percentile is remarkably poor (Figure 5—figure supplement 2).
These observations support the idea that the singletrial signals, ${S}^{\mathrm{ramp}}$, ${S}^{\text{PC1}}$, and ${S}_{\text{Tin}}^{\text{con}}$, approximate the DV used by the monkey to make its decision. In Figure 5—figure supplement 3, we show that the ${S}^{\text{What}}$ and ${S}^{\text{When}}$ coding directions achieve qualitatively similar results. Moreover, a late sample from ${S}^{x}(t)$ mediates the earlier correlation with RT and choice of signals rendered by other coding directions, ${S}^{y}(t)$, at earlier times. Such crossmediation is consistent with the high degree of cosine similarity between the coding directions (Figure 4d). The observation suggests that the DV is a prominent signal in LIP, discoverable by a variety of strategies, and consistent with the idea that it is onedimensional. In Figure 4—figure supplement 4, we show that linear and nonlinear decoders achieve similar performance, which argues against a nonlinear embedding of the DV in the population activity.
Activity of directionselective neurons in area LIP resembles momentary evidence
Up to now, we have focused our analyses on resolving the DV on single trials, paying little attention to how it is computed or to other signals that may be present in the LIP population. The driftdiffusion signal approximates the accumulation, or integral, of the noisy momentary evidence—a signal approximating the difference in the firing rates of directionselective (DS) neurons with opposing direction preferences (e.g, in area MT; Britten et al., 1996). DS neurons, with properties similar to neurons in MT, have also been identified in area LIP (Freedman and Assad, 2006; Shushruth et al., 2018; Fanini and Assad, 2009; Bollimunta and Ditterich, 2012), where they are proposed to play a role in motion categorization (Freedman and Assad, 2011). We hypothesize that such neurons might participate in routing information from DS neurons in MT/MST to those in LIP that contain a choice target in their response fields.
We identified such DS neurons using a passive motion viewing task (Figure 6a and b, left). Neurons preferring leftward or rightward motion constitute 5–10% of the neurons in our sample populations Table 1. Figure 6 shows the average firing rates of 51 leftwardpreferring neurons (${\text{M}}_{\text{in}}^{\text{left}}$, Figure 6a) and 26 rightwardpreferring neurons (${\text{M}}_{\text{in}}^{\text{right}}$, Figure 6b) under passive motion viewing and decisionmaking. The separation of the two traces in the passive viewing task is guaranteed because we used this task to identify the DS neurons. It is notable, however, that the DS is first evident about 100 ms after the onset of random dot motion, and this latency is also apparent in mean firing rates grouped by signed coherence during decision making (Figure 6a and b right). The activity of DS neurons is modulated by both the direction and strength of motion. However, unlike the ${\text{T}}_{\text{in}}$ neurons, the traces associated with different motion strengths are mostly parallel to one another and do not reach a common level of activity before the saccadic eye movement (i.e., they do not signal decision termination).
In addition to their shorter onset latency, the directionselectivity of ${\text{M}}_{\text{in}}$ neurons precedes the choiceselectivity of ${\text{T}}_{\text{in}}^{\text{con}}$ neurons by ∼100 ms (Figure 6c). The responses bear similarity to DS neurons in area MT. Such neurons are known to exhibit choicedependent activity insofar as they furnish the noisy evidence that is integrated to form the decision (Britten et al., 1996; Shadlen et al., 1996). We computed putative single trial direction signals by averaging the responses from the left and rightpreferring DS neurons, respectively. The resulting signals, ${S}_{\text{Min}}^{\text{left}}(t)$ and ${S}_{\text{Min}}^{\text{right}}(t)$, have weak leverage on choice, but the leverage does not increase as a function of time (Figure 6d, left). This is what would be expected if the ${\text{M}}_{\text{in}}$ neurons represent the noisy momentary evidence as opposed to the accumulation thereof (Mazurek et al., 2003). We failed to detect a correlation between RT and either ${S}_{\text{Min}}$ signal (Figure 6d, right). This is surprising, but it could be explained by lack of power—a combination of small numbers of ${\text{M}}_{\text{in}}$ neurons, narrow sample windows (50 ms boxcar) and the focus on the long RT trials. Indeed, we found a weak but statistically significant negative correlation between RT and the difference in leftward vs. rightward signals, averaged over the epoch $0.1\le t\le 0.4\phantom{\rule{thinmathspace}{0ex}}s$ from motion onset ($\mathrm{p}=0.0004;{\mathcal{H}}_{0}$ : ρ ≥ 0, see ‘Methods’).
We considered the hypothesis that these DS signals are integrated by the $\mathrm{T}}_{\mathrm{i}\mathrm{n}}^{\mathrm{c}\mathrm{o}\mathrm{n}$ neurons to form the DV. The heatmap in Figure 6e supports this hypothesis. On each trial, we formed the ordered pairs, $\{x,y\}$, where $x={\stackrel{~}{S}}_{\mathrm{M}\mathrm{i}\mathrm{n}}^{\mathrm{l}\mathrm{e}\mathrm{f}\mathrm{t}}({t}_{x}){\stackrel{~}{S}}_{\mathrm{M}\mathrm{i}\mathrm{n}}^{\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}}({t}_{x})$ and $y={\stackrel{~}{S}}_{\mathrm{T}\mathrm{i}\mathrm{n}}^{\mathrm{c}\mathrm{o}\mathrm{n}}({t}_{y})$. The tilde in these expressions indicates the use of standardized residual values, for each signed motion strength. The heatmap shows the correlation of these quantities across trials. If the hypothesis were true, the correlations should be positive for ${t}_{y}>{t}_{x}$ when ${t}_{x}>100$ ms and ${t}_{y}>200$ ms, and if the operation approximates integration, the level of correlation should be consistent at all lags, ${t}_{y}{t}_{x}>100$ ms. The correlations are significant in the epoch of interest, and they differ significantly from the average correlations in the rest of the graph (i.e., ${t}_{x}<100$, ${t}_{y}<200$, or $t}_{y}<{t}_{x$, p<0.0001 permutation test). Although correlative, the observation is consistent with the idea that evidence integration occurs within area LIP, rather than inherited from another brain area (Zhang et al., 2022; Bollimunta and Ditterich, 2012).
Discussion
We have observed a neural representation of the stochastic process that gives rise to a single decision. This is the elusive driftdiffusion signal that has long been thought to determine the variable choice and response time in the perceptual task studied here. The signal was elusive because it is the integral of noisy momentary evidence, hence stochastic, and undetectable in the firing rates when they are computed as averages over trials. The averages preserve the ramplike drift component, leaving open the possibility that the averages are composed of other stochastic processes (e.g., Latimer et al., 2015; Cisek et al., 2009). By providing access to populations of neurons in LIP, macaque Neuropixels probes (Trautmann et al., 2023) allowed us to resolve, for the first time, the evolution of LIP activity during a single decision.
The present findings establish that the ramplike averages arise from driftdiffusion on single trials, and this driftdiffusion signal approximates the DV that arbitrates the choice and RT on that trial. We used a variety of strategies to assign a weight to each neuron in the population such that the vector of weights defines a coding direction in NSS. The weighted averages render population firing rate signals on single trials. Our experience is that any method of assigning the weights that captures a known feature of evidence accumulation (or its termination in a saccadic choice) reveals driftdiffusion on single trials, and this also holds for datadriven, hypothesisfree methods such as PCA. This is because the actual dimensionality of the DV is effectively one—a scalar function of time that connects the visual evidence to a saccadic choice (Ganguli et al., 2008). Thus a weighting established by training a decoder at time $t=\tau $ to predict the monkey’s choice performs nearly as well when tested at times other than the time the decoder was trained on (i.e., $t\ne \tau $; Figure 4).
The different strategies for deriving coding directions lead to different weight assignments, but the coding directions are linearly dependent (Figure 4d). They produce traces, $S(t)$, that are similar (Figure 4e) and suggestive of driftdiffusion. Traces accompanying trials with the same motion coherence meander and spread apart at a rate similar to diffusion (i.e., standard deviation proportional to $\sqrt{t}$), and they exhibit a pattern of autocorrelation, as a function of time and lag, consistent with diffusion (Figure 3). The calculations applied in the present study improve upon previous applications (e.g., Churchland et al., 2011; de Lafuente et al., 2015; Shushruth et al., 2018) by incorporating the contribution of the smoothing to the autocorrelations. The departures from theory are explained by the fact that the accumulations are bounded. The upper bound and the fact that spike rates must be nonnegative (a de facto lower reflecting bound) limits the spread of the singletrial traces.
The singletrial signals, ${S}_{i}^{x}(t)$, approximate the DV that gives rise to the choice and decision time on trial $i$ (Figure 5, Figure 5—figure supplement 3). Support for this assertion is obtained using a conservative assay, which quantifies the leverage of the first 300 ms of the signal’s evolution on decision outcomes—choice and RT—occurring at least 670 ms after motion onset. Naturally, the signals do not explain all the variance of these outcomes. The sample size is limited to $N$ randomly selected, often weakly correlated neurons. The sample size and correlation are especially limiting for the ${\text{T}}_{\text{in}}^{\text{con}}$ neurons ($\mathbb{E}(r)=0.067\pm 0.0036$). Control analyses show that the degree of leverage on behavior and mediation of these relationships by later activity is on par with that obtained from simulated, weakly correlated neurons (Figure 5—figure supplement 1). In addition, because they are identified post hoc, many have response fields that barely overlap the choice target. Presumably, that is why their responses are weak compared to previous singleneuron studies in which the choice targets were centered in the response field by the experimenter. Yet even this noisy signal, ${S}_{\text{Tin}}^{\text{con}}$, mediates signals produced by coding directions using the entire population (Figure 5).
The ${\text{T}}_{\text{in}}^{\text{con}}$ neurons were the first to be identified as a plausible candidate neural representation of the DV, based on firing rate averages (Shadlen and Newsome, 1996; Platt and Glimcher, 1999). This neural type is also representative of the LIP projection to the region of the SC that represents the saccadic vector required to center the gaze on the choice target. (Paré and Wurtz, 1997). In a companion study by Stine et al., 2023 we show that the SC is responsible for cessation of integration in LIP. Features of the driftdiffusion signal from the ${\text{T}}_{\text{in}}^{\text{con}}$ neurons are correlated with bursting events in corresponding populations of ${\text{T}}_{\text{in}}^{\text{con}}$ neurons in the SC, including the final saccadic burst that ends the decision with a saccade to the contralateral choice target. Stine et al., 2023 also show that inactivation of ${\text{T}}_{\text{in}}^{\text{con}}$ neurons in the SC has little effect on ${S}_{\text{Tin}}^{\text{con}}$ signals in LIP.
Previous studies of LIP using the random dot motion task focused primarily on the ${\text{T}}_{\text{in}}^{\text{con}}$ neurons (cf. Meister et al., 2013). It was thus unknown whether and how other neurons contribute to the decision process. The Neuropixels probes used in the present study yield a large and unbiased sample of neurons. Many of these neurons have response fields that overlap one of the two choice targets, but the majority have response fields that overlap neither the choice targets nor the random dot motion. Our screening procedures (delayed saccades and passive motion viewing tasks) do not supply a quantitative estimate of their spatial distribution. It is worth noting that neurons with response fields that overlap neither of the two choice targets were assigned nonzero weights by the What and Whendecoders, and yet, removal of the taskrelated neurons that represent the choice targets and motion (i.e., ${\text{T}}_{\text{in}}$ and ${\text{M}}_{\text{in}}$) decreases decoding accuracy more substantially than removing all but the ${\text{T}}_{\text{in}}$ neurons (Figure 4—figure supplement 5). The accuracy the decoder achieves is likely explained by neurons with weak responses that simply failed to meet our criterion for inclusion in the ${\text{T}}_{\text{in}}$ and ${\text{M}}_{\text{in}}$ categories (e.g., neurons with response fields that barely overlap the choice targets or RDM). Some neurons outside these groups might reflect normalization signals from the ${\text{T}}_{\text{in}}$ and ${\text{M}}_{\text{in}}$ neurons (Shushruth et al., 2018; Carandini and Heeger, 2011), imbuing broad, decisionrelated covariability across the population. It thus seems possible that higher dimensional tasks (e.g., four choices instead of two) could decrease correlations among groups of neurons with different response fields.
The fact that the raw averages from a small number of weakly correlated ${\text{T}}_{\text{in}}$ neurons furnish a DV on par with that furnished by the full population underscores the importance of this functional class. The role of the ${\text{M}}_{\text{in}}$ neurons is less well understood. Freedman and colleagues described direction selective neurons in LIP, similar to our ${\text{M}}_{\text{in}}$ neurons (Freedman and Assad, 2011; Fanini and Assad, 2009; Sarma et al., 2016). They showed that the neurons represent both the direction of motion and the decision in their task. In contrast, we do not observe the evolution of the decision (i.e., DV) by the ${\text{M}}_{\text{in}}$ neurons (Figure 6). The latency of the direction and coherencedependent signal as well as its dynamics resemble properties of DS neurons in area MT. The delayed correlation between ${\text{M}}_{\text{in}}$ and ${\text{T}}_{\text{in}}$ responses evokes the intriguing possibility that ${\text{M}}_{\text{in}}$ neurons supply the momentary evidence, which is integrated within LIP itself (Zhang et al., 2022). Future experiments that better optimize the yield of ${\text{M}}_{\text{in}}$ neurons will be informative, and direct, causal support will require perturbations of functionally identified ${\text{M}}_{\text{in}}$ neurons, which is not yet feasible. A natural question is why LIP would contain a copy of the DS signals that are already present in area MT. We suspect it simplifies the routing of momentary evidence from neurons in MT/MST to the appropriate ${\text{T}}_{\text{in}}$ neurons. This interpretation leads to the prediction that DS ${\text{M}}_{\text{in}}$ neurons would be absent in LIP of monkeys that are naïve to saccadic decisions informed by random dot motion, as has been observed in the SC (Horwitz et al., 2004). Further, when motion is not the feature that informs the saccadic response—for example, in a color categorization task (e.g., Kang et al., 2021)—LIP might contain a representation of momentary evidence for color (Toth and Assad, 2002; Sereno and Maunsell, 1998).
The capacity to record from many neurons simultaneously invites characterization of the population in NSS, in which the activity of each neuron defines a dimension. Often, population activity is confined to a lowdimensional subspace or manifold within the NSS (Vyas et al., 2020). An evermorepopular viewpoint is that representations within these subspaces are emergent properties of the population—that is, distributed, rather than coded directly by single neurons—a dichotomy that has its roots in Barlow’s neuron doctrine (as updated in Barlow, 1994). Indeed, it is tempting to conclude that the driftdiffusion signal in LIP is similarly emergent based on our NSS analyses—the identified subspaces (i.e., coding directions) combine neurons with highly diverse activity profiles. In contrast, grouping neurons by the location of their spatial response field reveals a direct coding scheme: ${\text{T}}_{\text{in}}$ neurons directly represent the accumulated evidence for making a particular saccade and ${\text{M}}_{\text{in}}$ neurons represent the momentary evidence. We argue that this explanation is more parsimonious and, importantly, more principled. Grouping neurons based on spatial selectivity rests on the principle that neurons with similar RFs have similar projections, which is the basis for topographic maps in the visual and oculomotor systems (Schall, 1995; Silver and Kastner, 2009; Kremkow et al., 2016; Felleman and Van Essen, 1991). In contrast, there are no principles that guide the grouping of neurons in state space analyses, as the idea is that they may comprise as many dimensions as there are neurons that happen to be sampled by the recording device.
The present finding invites both hope and caution. It may be useful to consider a counterfactual state of the scientific literature that lacks knowledge of the properties of LIP ${\text{T}}_{\text{in}}$ neurons—a world without Gnadt and Andersen, 1988 and no knowledge of LIP neurons with spatially selective persistent activity. In this world we have no reason to entertain the hypothesis that decisions would involve neurons that represent the choice targets. We do know about DS neurons in area MT and their causal role in decisions about the direction of random dot motion (Salzman et al., 1992; Fetsch et al., 2018; Ditterich et al., 2003; Liu and Pack, 2017). We also know that driftdiffusion models explain the choiceresponse time behavior. Guided by no particular hypothesis, we obtain population neural recordings in the random dot motion task. We do not perform the saccade and passive viewing control experiments. What might we learn from such a dataset? We might apply PCA and/or train a choice decoder or possibly a Whendecoder. If so, we could discover the driftdiffusion signal and we might also infer that the dimensionality of the signal is low. However, we would not discover the ${\text{T}}_{\text{in}}$ neurons without a hypothesis and a test thereof. We might notice that the coding directions that reveal driftdiffusion often render a response at the onset of the choice targets as well as increased activity at the time of saccades to the contralateral choice target. These facts might lead us to hypothesize that the population might contain neurons with visual receptive fields and some relationship to saccadic eye movements. We might then query individual neurons, post hoc, for these features, and ask if they render the driftdiffusion signal too. The inferences could then be tested experimentally by including simple delayed saccades in the next experiment. The hope in this counterfactual is that datadriven, hypothesisfree methods can inspire hypotheses about the mechanism. The caution is to avoid the natural tendency to stop before the hypotheses and tests, thus accepting as an endpoint the characterization of population dynamics in high dimensions or a lower dimensional manifold. If LIP is representative, these mathematically accurate characterizations may fail to illuminate the neurobiological parsimony.
Methods
Ethical approval declarations
Two adult male rhesus monkeys (Macaca mulatta, Primate Products) were used in the experiments. All training, surgery, and experimental procedures complied with guidelines from the National Institutes of Health and were approved by the Institutional Animal Care and Use Committee at Columbia University (protocols AAAN4900 and ACAAAW4454). A head post and two recording chambers were implanted under general anesthesia using sterile surgical procedures (for additional details, see So and Shadlen, 2022). One recording chamber allowed access to area LIP in the right hemisphere. The other was placed on the midline, allowing access to the SC. Those recordings are described in Stine et al., 2023. Here we report only on the neural recordings from LIP, focusing on the epoch of decision formation.
Behavioral tasks
The monkeys were trained to interact with visual stimuli presented on a CRT video monitor (Vision Master 1451, Iiyama; viewing distance 57 cm; frame rate 75 Hz) using the psychophysics toolbox (Brainard, 1997; Pelli, 1997; Kleiner et al., 2007). Task events were controlled by Rex software (Hays et al., 1982). The monkeys were trained to control their gaze and make saccadic eye movements to peripheral targets to receive a liquid reward (juice). The direction of gaze was monitored by an infrared camera (EyeLink 1000; SR Research, Ottawa, Canada; 1 kHz sampling rate). The tasks involve stages separated by random delays, distributed as truncated exponential distributions
where t_{min} and t_{max} define the range, $\lambda $ is the time constant, and $\alpha $ is chosen to ensure the total probability is unity. Below, we provide the range (${t}_{\text{min}}$ to ${t}_{\text{max}}$) and the exponential parameter $\lambda $ for all variable delays. Note that because of truncation, the expectation $\mathbb{E}(t)<{t}_{\text{min}}+\lambda $.
In the main task (Figure 1a), the monkey must decide the net direction of random dot motion and indicate its decision when ready by making a saccadic eye movement to the corresponding choice target. After acquiring a central fixation point and a random delay (0.25–0.7 s, $\lambda =$ 0.15), two red choice targets (diameter 1 dva) appear in the left and right visual fields. The random dot motion is then displayed after a random delay (0.25–0.7 s, $\lambda =$ 0.4 s) and continues until the monkey breaks fixation. The dots are confined to a circular aperture (diameter 5 dva; degrees visual angle) centered on the fixation point (dot density 16.7 dots⋅dva^{2}s^{1}). The direction and strength of motion are determined pseudorandomly from $\pm \{0,3.2,6.4,12.6,25.6,51.2\}$% coherence. The sign of the coherence indicates direction (positive for leftward, which is contraversive with respect to the recorded hemisphere). The absolute value of coherence determines the probability that a dot plotted on frame $n$ will be displaced by $\mathrm{\Delta}x$ on frame $n+3$ ($\mathrm{\Delta}t=40\mathrm{m}\mathrm{s}$), as opposed to randomly replaced, where $\mathrm{\Delta}x=\frac{5}{75}$ dva, consistent with 5 dva⋅s^{1} speed of apparent motion (see also Roitman and Shadlen, 2002). The monkey is rewarded for making a saccadic eye movement to the appropriate choice target. On trials with 0% coherent motion, either saccadic choice is rewarded with probability 1/2. Errors are punished by extending the intertrial interval by up to 3 s (see Stine et al., 2023, for additional details). On approximately half of the trials, a 100 ms pulse of weak motion (±3.2% or 4.0% coherence for monkeys J and M, respectively) is added to the random dot motion stimulus at a random time (0.1–0.8 s, $\lambda =$ 0.4) relative to motion onset (similar to Kiani et al., 2008). Monkey M performed 9684 trials (five sessions); monkey J performed 8142 trials (three sessions). The data are also analyzed in a companion paper that focuses on the termination of the decision (Stine et al., 2023).
In the visually instructed delayed saccade task (Hikosaka and Wurtz, 1983), one target is displayed at a pseudorandom location in the visual field. After a variable delay (monkey M: 0.4–1.1 s, $\lambda =$ 0.3; monkey J: 0.5–1.5 s, $\lambda =$ 0.2), the fixation point is extinguished, signaling ‘go’. The monkey is rewarded for making a saccade to within ±2.5 dva of the location of the target. In a memoryguided variant of the task (Gnadt and Andersen, 1988; Funahashi et al., 1989), the target is flashed briefly (200 ms) and the monkey is required to make a saccade to the remembered target location when the fixation point is extinguished. These tasks provide a rough characterization of the neural response fields during the visual, perisaccadic and delay epochs. Neurons are designated ${\text{T}}_{\text{in}}^{\text{con}}$ if they exhibit spatially selective activity at the location of the response target in the visual hemifield contralateral to the recorded hemisphere. This determination is made before analyzing the activity in the random dot motion task. We refer to the unweighted mean firing rate as ${S}_{\text{Tin}}^{\text{con}}$. Neurons are designated ${\text{T}}_{\text{in}}^{\text{ips}}$ if they exhibit spatially selective activity at the location of the response target in the visual hemifield ipsilateral to the recorded hemisphere. These analyses were conducted post hoc, after spike sorting.
The passive motionviewing task is identical to the main task, except there are no choice targets and only the strongest motion strength (±51.2% coherence) is displayed for 500 ms (1 s on a small fraction of trials in session 1). The direction is left or right, determined randomly on each trial (${P}_{\text{left}}=\frac{1}{2}$). The monkey is rewarded for maintaining fixation until the random dot motion is extinguished.
Behavioral analyses
We fit a neurally inspired variant of the driftdiffusion model (Figure 1c) to the choiceRT data from each session. The model constructs the decision process as a race between two accumulators: one accumulating evidence for left and against right (e.g., left minus right) and one accumulating evidence for right and against left (e.g., right minus left). The decision is determined by the accumulator that first exceeds its positive decision bound, at which point the decision is terminated. The races are negatively correlated with one another, owing to the common source of noisy evidence. We assume they share half the variance, $\rho =\sqrt{0.5}\approx 0.71$, but the results are robust to a wide range of reasonable values. The decision bounds are allowed to collapse linearly as a function of time, such that
We used the method of images (van den Berg et al., 2016; Shan et al., 2019) to compute the probability density of the accumulated evidence for each accumulator (which both start at zero at $t=0$) as a function of time ($t$) using a time step of 1 ms. The decision time distributions rendered by the model were convolved with a Gaussian distribution of the nondecision times, ${t}_{\text{nd}}$, which combines sensory and motor delays, to generate the predicted RT distributions. The model has six parameters: $\kappa ,{B}_{0},\alpha ,{\mu}_{\text{nd}},{\sigma}_{\text{nd}}$, and ${C}_{0}$, where $\kappa $ determines the scaling of motion strength to drift rate, ${C}_{0}$ implements bias in units of signed coherence (Hanks et al., 2011), ${\mu}_{\text{nd}}$ is the mean nondecision time, and ${\sigma}_{\text{nd}}$ is its standard deviation (Table 2). Additional details about the model and the fitting procedure are described in van den Berg et al., 2016.
Simulated decision variables
We fit the race model described above to the combined behavioral data across all sessions (separately for each monkeys) and used the bestfitting parameters for monkey M (see Table 2) to simulate a total of 60,000 trials representing all signed coherences of the motion discrimination task. Each simulated trial yields a time series for two DVs, one for each accumulator in the race. We assume that the modelderived nondecision time (${t}_{\text{nd}}=317$ ms; Figure 1b) comprises visual and motor processing times at the beginning and end of the decision: 200 ms from motion onset to the beginning of evidence integration, and the remaining 117ms after termination. The latter approximates the variability observed in the saccadic latencies in the delayed saccade task and is simulated using a normal distribution, $\mathcal{N}(\mu ,\sigma )$, where $\mu =117$ ms and $\sigma =39$ ms (Stine et al., 2023). In this variable time period between decision termination and the response (saccade), the simulated DVs were assigned the values they had attained at the start of this epoch. For all analyses that employ these simulations, we use the DV for the leftchoice accumulator because the neural recordings were from LIP in the right hemisphere.
Neurophysiology
We used prototype ‘alpha’ version Neuropixels1.0NHP45 probes (IMEC/HHMIJanelia) to record the activity of multiple isolated single units from the ventral subdivision of area LIP (LIP_{v}; Lewis and Van Essen, 2000). We used anatomical MRI to identify LIP_{v} and confirmed its physiological hallmarks with singleneuron recordings (Thomas Recording GmbH) before proceeding to multineuron recordings. Neuropixels probes enable recording from 384 out of 4416 total electrical contacts distributed along the 45mmlong shank. All data presented here were recorded using the 384 contacts closest to the tip of the probe (Bank 0), spanning 3.84 mm. Reference and ground signals were directly connected to each other and to the monkey’s head post. A total of 1084 neurons were recorded over eight sessions (54–203 neurons per session) (Table 1).
The Neuropixels 1.0NHP45 probe uses a standard Neuropixels 1.0 headstage and is connected via the standard Neuropixels1.0 5m cable to the PCI eXtensions for Instrumentation (PXIe) hardware (PXIe1071 chassis and PXI6141 and PXIe8381 I/O modules, National Instruments). Raw data were acquired using the SpikeGLX software (http://billkarsh.github.io/SpikeGLX/), and single units were identified offline using the Kilosort 2.0 algorithm (Pachitariu et al., 2016; Pachitariu et al., 2020), followed by manual curation using Phy (https://github.com/cortexlab/phy).
Neural data analysis
The spike times from each neuron are represented as delta functions of discrete time, ${s}_{i,n}(t)$, on each trial $i$ and each neuron $n$ ($dt=1$ ms). The weighted sum of these ${s}_{i,n}(t)$ gives rise to the singletrial population signals:
where the superscript, $x$, identifies the method or source that establishes the weights—that is, the coding direction in NSS or the neuron type contributing to a pooled average (e.g., ${\text{T}}_{\text{in}}^{\text{con}}$). For visualization, the singletrial signals are smoothed by convolution with a truncated Gaussian using the MATLAB function, gausswin (width $=80$ ms, width factor $=1.5$, $\sigma \approx 26$ ms). Unless otherwise specified, all other analyses employ a 50 ms boxcar (rectangular) filter; values plotted at time $t$ include data from $t24\text{to}t+25$ ms.
We used several methods to define coding directions in the NSS defined by the population of neurons in each session. For PCA and choice decoding, we standardized the singletrial firing rates for each neuron using the mean and standard deviation of its firing rate at in the epoch $200\le t\le 600$ ms after motion onset. This practice led to the exclusion of two neurons (session 1) that did not produce any spikes in the normalization window. Those neurons were assigned zero weight.
$\mathrm{T}}_{\mathrm{i}\mathrm{n}$ neurons
Neurons were classified post hoc as ${\text{T}}_{\text{in}}$ by visualinspection of spatial heatmaps of neural activity acquired in the delayed saccade task. We inspected activity in the visual, delay, and perisaccadic epochs of the task. The distribution of target locations was guided by the spatial selectivity of simultaneously recorded neurons in the SC (see Stine et al., 2023, for details). Briefly, after identifying the location of the SC response fields, we randomly presented saccade targets within this location and seven other, equally spaced locations at the same eccentricity. In monkey J, we also included 1–3 additional eccentricities, spanning 5–16 degrees. Neurons were classified as ${\text{T}}_{\text{in}}$ if they displayed a clear, spatially selective response in at least one epoch to one of the two locations occupied by the choice targets in the main task. Neurons that switched their spatial selectivity in different epochs were not classified as ${\text{T}}_{\text{in}}$. The classification was conducted before the analyses of activity in the motion discrimination task. The procedure was meant to mimic those used in earlier singleneuron studies of LIP (e.g., Roitman and Shadlen, 2002) in which the location of the choice targets was determined online by the qualitative spatial selectivity of the neuron under study. The ${\text{T}}_{\text{in}}^{\text{con}}$ neurons in the present study were highly selective for either the contralateral or ipsilateral choice target used in the RDM task ($\mathrm{A}\mathrm{U}\mathrm{C}=0.89\pm 0.01$ for 97% of neurons, Wilcoxon ranksum test). Given the sparse sampling of saccade target locations, we are unable to supply a quantitative estimate of the center and spatial extent of the RFs. We next describe the methods to establish the coding directions.
Ramp direction
We applied linear regression to generate a signal that best approximates a linear ramp, on each trial, $i$, that terminates with a saccade to the choice target contralateral to the hemisphere of the LIP recordings. The ramps are defined in the epoch spanning the decision time: each ramp begins at ${f}_{i}({t}_{0})=1$, where ${t}_{0}=0.2$ s after motion onset, and ends at ${f}_{i}({t}_{1})=1$, where ${t}_{1}={t}_{\text{sac}}0.05$ s (i.e., 50 ms before saccade initiation). The ramps are sampled every 25 ms and concatenated using all eligible trials to construct a long sawtooth function (see Figure 2—figure supplement 1). The regression solves for the weights assigned to each neuron such that the weighted sum of the activity of all neurons best approximates the sawtooth. We constructed a time series of standardized neural activity, sampled identically to the sawtooth. The spike times from each neuron are represented as delta functions (rasters) and convolved with a noncausal 25 ms boxcar filter. The mean and standard deviation of all sampled values of activity were used to standardize the activity for each neuron (i.e., Ztransform). The coefficients derived from the regression establish the vector of weights that define ${S}^{\mathrm{ramp}}$. The algorithm ensures that the population signal ${S}^{\mathrm{ramp}}(t)$, but not necessarily individual neurons, have amplitudes ranging from approximately 1 to 1.
We employed a lasso linear regression with $\lambda =0.005$. The vector of weights assigned across the neurons defines a direction in NSS, ${S}^{\mathrm{ramp}}$, which we use to render the signal ${S}^{\mathrm{ramp}}(t)$ on single trials by projecting the data onto this direction. To determine the effect of the regularization term in the lasso regression, we recomputed singletrial signals using standard linear regression, without regularization. We then calculated the Pearson correlation between singletrial traces generated by projecting neural data onto the two coding directions (i.e., with and without regularization). The high correlation between singletrial traces ($\text{mean}r=0.99\text{, across sessions}$) indicates that the findings are not a result of the regularization applied. Here and elsewhere we compute the mean $r$ using the Fisherz transform, such that
where ${Z}^{\text{inv}}$ is the inverse Fisherz.
Principal component analysis (PCA)
We applied standard PCA to the firing rate averages for each neuron using all trials sharing the same signed motion coherence in the shorter of two epochs: 200 ms to either 600 ms after motion onset or 100 ms before the median RT for the signed coherence, whichever produces the shorter interval. The results of the PCA indicate that the dimensionality of the data is low, but greater than 1. The participation ratio is 4.4 ± 1.3 (Mazzucato et al., 2016; Gao et al., 2017) and the first three PCs explain 67.1 ± 3.1% of the variance on average (mean ± s.e.m. across sessions). As in all other analyses of neural activity aligned to motion onset, we exclude data in the 100 ms epoch ending at saccade initiation on each trial. We projected the neural data onto the first PC to generate the signal ${S}^{\text{PC1}}$.
Choice decoder
For each experimental session, we trained logistic choice decoders with lasso regularization ($\lambda =0.01$) on the population activity in 50 ms time bins spanning the first 500 ms after motion onset and the 300 ms epoch ending at saccade initiation, respectively. Each of the decoders was trained on the evennumbered trials. Decoder accuracy was crossvalidated using the activity of heldout, odd trials at the same time point (Figure 4a). For the time bins aligned to motion onset, we also assessed the accuracy of the decoders trained on each of the time bins to predict the choice on time bins on which they were not trained (Figure 4b; King and Dehaene, 2014). We use the decoder trained on the bin centered on $t=450$ ms to define the What coding direction. We refer to this as the fixed trainingtime decoder to distinguish it from the standard machinelearning decoder, which assigns a potentially distinct vector of weights at each time point. We applied a similar analysis to simulated data (see ‘Simulated DVs’) to generate the black curve in Figure 4a. Assuming a stochastic driftdiffusion process giving rise to choice and response times, the exercise establishes a rough upper bound on decoder accuracy, were the actual driftdiffusion process known precisely.
When decoder
This decoder is trained to predict whether a saccade to the left (contralateral) choice target will occur within the next 150 ms. We applied logistic regression with lasso regularization ($\lambda =0.01$) to spike counts from each neuron in discrete bins of 25 ms, from 200 ms before motion onset to 50 ms before the saccade. We used only trials ending in a left choice (including errors) and trained the decoder on the even numbered half of those trials. The concatenation of these trials forms a sequence of step functions which are set to 1 if a saccade occurred within 150 ms of the start of the 25 ms time bin and 0 otherwise (Figure 4—figure supplement 1).
The spike counts were also concatenated across these trials to construct column vectors (one per neuron) that match the vector of concatenated step functions. These concatenated vectors, one per neuron, plus an offset (${\beta}_{0}$), serve as the independent variables of the regression model (one $\beta $ term per neuron). The proportion of $\beta $ weights equal to zero, controlled by the lasso parameter, $\lambda $, was $0.8\pm 0.02$ across sessions. The weights define the ${S}^{\text{When}}$ coding direction, which yields singletrial signals, ${S}^{\text{When}}(t)$. The Whendecoder signal is ${S}^{\text{When}}(t)+{\beta}_{0}$. We validated the Whendecoder by computing the area under an ROC (AUC) using the heldout (oddnumbered) trials ending in left choices (mean AUC over all time points and sessions: $0.84\pm 0.024$, mean ± s.e.).
Our motivation, however, was to ascertain whether the When coding direction also predicts the monkey’s choices on all trials—that is, to perform as a What decoder. To this end, we predicted the choice using the sign of the detrended ${S}^{\mathrm{W}\mathrm{h}\mathrm{e}\mathrm{n}}(t)$, formed by subtracting the average of the signal using all trials:
where ${\u27e8\mathrm{\cdots}\u27e9}_{i}$ denotes expectation across all trials contributing values at time $t$. The choice accuracy is
The green trace in Figure 4a shows ${\u27e8{A}_{i}(t)\u27e9}_{i}$.
Aggregation of data across experimental sessions
To combine singletrial data across sessions (e.g., ${S}^{x}(t)$), we first normalize activity within each session as follows. Using all trials ending in the same choice, we construct the trialaveraged activity aligned to both motion onset ($0\le {t}_{motion}\le 0.6$ s) and saccade onset ($0.6\le {t}_{sacc}\le 0$ s). This produces four traces. The minimum and maximum values (${a}_{min}$ and ${a}_{max}$) over all four traces establish the range, zero to one, of the normalized signal:
where lowercase ${\widehat{s}}^{x}(i,t)$ is the normalized signal across trials $i$ and time $t$ in an individual session.
Cosine similarity
We computed the cosine similarity between the weight vectors that define coding directions ${S}^{\mathrm{ramp}}$, ${S}^{\text{PC1}}$, ${S}_{\text{Tin}}^{\text{con}}$, ${S}^{\text{What}}$, and ${S}^{\text{When}}$. Mean cosine similarities are portrayed in the heatmap in Figure 4d and also in Figure 4—figure supplement 3, top, where they are accompanied by error bars. We evaluated the null hypothesis that the mean cosine similarity is ≤0 with ttests. We also performed two control analyses that deploy random coding directions in NSS. For each of the original coding directions, we obtained 1000 random coding directions as random permutations of the weight assignments. The cosine similarities between pairs of such random directions in state space are shown in Figure 4—figure supplement 3, top. The cumulative distribution of cosine similarities under permutation supports pvalues less than 1/1000. In a second control analysis, we used random unit vectors as random coding directions (normal distribution with mean 0 and scaled to unit length).
Similarity of singletrial signals
We calculated Pearson correlation to quantify the similarity of the signals generated by pairs of coding directions, $x$ and $y$. For each trial, i, the detrended signals, ${\stackrel{~}{S}}_{i}^{x}$ and ${\stackrel{~}{S}}_{i}^{y}$, provide ordered pairs, $\{{\stackrel{~}{S}}_{i}^{x}({t}_{j}),{\stackrel{~}{S}}_{i}^{y}({t}_{j})\}$ where $j$ indexes successive 50 ms bins between 200 ms after motion onset and 100 ms before saccade initiation. We excluded trials comprising less than four such bins. Each trial gives rise to a correlation coefficient, ${r}_{i}$. We report the mean $r$ using Equation 4. The $r$values for comparisons across all pairs of coding directions are summarized in Figure 4e, and variability across sessions is portrayed in Figure 4—figure supplement 3, bottom. For each pair of CDs and session, we evaluated the null hypothesis, ${\mathscr{H}}_{0}:\overline{r}\le 0$ (ttest).
We also performed two control analyses that deploy random CDs in NSS. These analyses control for the possibility that the correlations observed in the signals are explained by pairwise correlations between the neurons, regardless of the signals produced by the weighted sums. (i) We generated sets of singletrial traces ${S}^{\text{rand}}(t)$ by projecting the neural responses onto random CDs, defined by permuting the weights of each coding direction (${S}^{\mathrm{ramp}}$, ${S}^{\text{PC1}}$, ${S}_{\text{Tin}}^{\text{con}}$, ${S}^{\text{What}}$, and ${S}^{\text{When}}$). For each pair of CDs, we compute withintrial correlations between ordered pairs of trials using the same method applied to the original signals. We repeat this process for a total of 1000 random permutations per pair of CDs, per session. (ii) We sample a pair of random weight vectors from standard normal distributions. Each weight vector has a dimension equal to the number of recorded neurons in the session. The weight vectors are normalized to sum to 1. We generate random CDs using these weights and compute withintrial correlations using the same method applied to the original signals. We repeat this process 1000 times per session. For both analyses, we evaluated the null hypothesis that the observed correlations are not greater than those produced by the random projections (ttests using the Fisherz transformed correlations). Mean ± stdev of the mean rvalues between ordered pairs for both control analyses are summarized in Figure 4—figure supplement 3, bottom.
Leverage of singletrial activity on behavior
The leverage of singletrial signals, ${S}^{x}(t)$, on choice and RT was assessed using the earliest 300 ms epoch of putative integration ($0.2<t<0.5$ s from motion onset), restricting analyses to trials with RTs outside this range ($0.67<\text{RT}<2$ s). The singletrial signals are smoothed with a 50 ms boxcar filter and detrended by subtracting the mean $S(t)$ for trials sharing the same motion strength and direction (i.e., signed coherence). The RTs are also expressed as residuals relative to the mean RT, using trials sharing the same signed coherence and choice. We include trials with $\text{coh}\le 0.064$ that result in choices of the left response target in this analysis. Including trials of $\text{coh}\le 0.128$ produced comparable results. The leverage on RT is the Pearson correlation between the residual signals $\stackrel{~}{S}(t)$ at each time $0.2<t\le 0.5$, and $\stackrel{~}{RT}$ on that trial, where the tilde indicates residual. Correlations were computed per session and then averaged across sessions (Equation 4). We also show the correlation at t = 0.55 s, using the ordered pairs, $\{\stackrel{~}{S}(t=0.55),\stackrel{~}{{\textstyle \text{RT}}}\}$. We quantify mediation of the leverage of earlier samples by the later sample of ${S}^{x}$ by computing partial correlations $\{\stackrel{~}{S}(t),\stackrel{~}{{\textstyle \text{RT}}}\mid \stackrel{~}{S}(t=0.55)\}$, also notated ${R}_{\stackrel{~}{S}(t),\stackrel{~}{RT}\mid \stackrel{~}{S}(0.55)}$ in Figure 5. We show this mediation at all time points. We also report a mediation statistic (${\xi}^{\text{RT}}$) using the time point 200 ms after the beginning of putative integration (i.e., $S(t=0.4)$):
The rationale for using the 400 ms time point is (i) to allow the process to have achieved enough leverage on RT so that a reduction is meaningful and (ii) to preserve a substantial gap between this time and the sample at $t=0.55$ s (e.g., to avoid autocorrelations imposed by smoothing). The rare cases in which there was no negative correlation between $S(t=0.4)$ and RT were excluded from this summary statistic, ${\xi}^{\text{RT}}$, because no mediation is possible (Session 2: ${S}^{\mathrm{ramp}}$ & ${S}^{\text{PC1}}$). When combining values of ${\xi}^{\text{RT}}$ across sessions, we rectify any ${R}_{\stackrel{~}{S}(t),\stackrel{~}{RT}\mid \stackrel{~}{S}(0.55)}>0$ to zero. This occurs rarely when the mediated correlation is near zero, typically at early times.
We compute the leverage on choice, ${\xi}^{\text{Ch}}$, using trials with $\text{coh}\le 0.064$ and the same time points as for ${\xi}^{\text{RT}}$. Instead of an Rsquared measure, we based ${\xi}^{\text{Ch}}$ on coefficients derived from logistic regression:
where ${\beta}_{0}^{\text{coh}}$ is a set of constants that accounts for the proportion of contralateral choices at each signed motion strength and ${\beta}_{1}(t)$ is the simple leverage of $S(t)$ on choice, analogous to simple correlation. The regression analysis was performed separately for each session. The coefficients ${\beta}_{1}(t)$ were divided by their standard error and then averaged across sessions. This normalization step was implemented to control for potential variation in the magnitude of $S(t)$ (and therefore of ${\beta}_{1}(t)$) across sessions. Analogous to partial correlation, we include the later time point and fit
where ${\beta}_{1}^{*}(t)$ is the amount of leverage at time $t$ given the $S(0.55)$. The regression coefficients ${\beta}_{1}^{*}(t)$ were averaged across sessions after dividing them by the standard error of the ${\beta}_{1}(t)$ coefficients obtained from Equation 8. That is, the same normalization factors were used for the mediated and unmediated leverage. The summary statistic for choice mediation is defined by
For both types of mediation, we also test whether the earlier $S(t)$ is mediated by the ${\text{T}}_{\text{in}}^{\text{con}}$ neurons by substituting ${S}_{\text{Tin}}^{\text{con}}(0.55)$ for $S(0.55)$ in Equation 9 and in the expression for partial correlations.
Because the mediation statistics, ${\xi}^{\text{RT}}$ and ${\xi}^{\text{Ch}}$, are, by their definition, nonnegative, we assess statistical significance by bootstrapping. For each session, we construct 1000 surrogate data sets equal in size to the original data by sampling with replacement. The standard deviation of the leverage and mediation values at each time approximates the standard error. We compare the distribution of the mediation statistics, $\zeta $, to their distribution under the null hypothesis, ${\mathscr{H}}_{0}$, that the values arise by chance, instantiated by breaking the correspondence with the trial giving rise to the later sample, ${S}^{x}(t=0.55s)$. The permutation maintains correspondence in signed motion coherence. We compare the distributions of mediation from the bootstrap and ${\mathscr{H}}_{0}$ using the Wilcoxon ranksum test.
To test whether the observed leverage of neural activity on choice and RT is achieved by projections onto arbitrary coding directions, we generated random weight vectors by permuting the weights associated with the first PC for each session. We projected activity onto this random coding direction, applied the mediation analyses described above to this signal, and repeated this process 1000 times to produce a null distribution at each time point. The reported pvalues represent the probability that the observed leverage was generated from this null distribution.
We performed a similar analysis to test whether the observed leverage depends on the trialtotrial correspondence between neural activity and behavior. Here, the null distribution at each time point was generated by randomly permuting the trial indices associated with the neural activity and those associated with the behavioral measures.
Finally, we used the simulated data from the racing accumulator model to test the degree of leverage and mediation expected had we known the groundtruth DV on each trial (Figure 5—figure supplement 1). Because the simulated DV is noiseless (and the process is Markovian), the mediation is expected to be complete for all time points tested in the analyses. We therefore took two steps to make the simulated data more comparable to neural data: (i) we subsampled the simulated data to match the number of trials in each session. (ii) We generated $N$ noisy instantiations of the signal for each of the subsampled, simulated trials, where $N$ is the number of ${\text{T}}_{\text{in}}^{\text{con}}$ neurons in each session. The added noise is independent across time points and weakly correlated across all $N\times (N1)/2$ neuron pairs ($r\approx 0.09$). We then applied the mediation analyses to the mean of these signals and repeated this process 1000 times.
We performed three control analyses to determine whether the results of the mediation analysis were specific, meaningful, and comparable to the results obtained for the DV of a race model. To showcase that the results are specific, we generated random weight vectors by permuting the weights of the PC1 coding direction. We repeated this procedure 1000 times per session and projected the data along these directions in NSS to generate ${S}^{\text{rand}}$. We then computed the mediation analyses detailed above on these signals and determined significance by comparing the leverage on choice and correlation with RT of ${S}^{\mathrm{ramp}}$, ${S}^{\text{PC1}}$, and ${S}_{\text{Tin}}^{\text{con}}$ to the null distributions of ${S}^{\text{rand}}$ at each time point.
To estimate an upper limit for the degree of possible leverage and mediation, we simulated 60,000 trials using the race model that best fits the behavioral data of monkey M (see ‘Simulated DVs’). For any noisefree representation of a Markovian integration process, the leverage of an early sample of the DV on behavior would be mediated completely by later activity as the latter sample by definition encompasses all variability captured by the earlier sample. We, therefore, took two steps to make the simulated DVs more comparable to real neural data. (i) For each session, we first subsampled the simulated data to match the each session. (ii) To evaluate a DV approximated from the activity of $n{\text{T}}_{\text{in}}^{\text{con}}$ neurons per session rather than the true DV represented by the entire population, we then generated $n$ noisy instantiations of the signal for each simulated trial. The added noise is independent across time points and weakly correlated across neurons ($r\approx 0.09$). We then computed the measured DV ${S}^{\text{sim}}$ as the mean activity of these $n$ simulated neurons. We repeated this procedure 1000 times per session. Figure 5—figure supplement 1, bottom, displays the mean and standard deviation across permutations of the leverage of ${S}^{\text{sim}}(t)$ on behavior. The simulation results highlight that we would not expect the mediation of the leverage on behavior by a later sample to be complete (i.e., zero mediated leverage for all t < 0.55).
Noise correlation between neurons
The mean pairwise correlation between ${\text{T}}_{\text{in}}^{\text{con}}$ neurons, reported in ‘Results’, is based on all pairs of simultaneously recorded ${\text{T}}_{\text{in}}^{\text{con}}$ neurons in each session and all trials with $RT>0.5$ s. For each neuron and each trial, we compute the timeaveraged activity over the epoch $0.2\le t\le 0.4$. These scalar values are converted to residuals by subtracting the mean (for each neuron) across all trials sharing the same signed motion coherence. The residuals from all eligible trials are concatenated for each neuron to support the calculation of $N\times (N1)$ Pearson $r$ values, where N is the number of ${\text{T}}_{\text{in}}^{\text{con}}$ neurons in the session. The mean correlation for all pairs of ${\text{T}}_{\text{in}}^{\text{con}}$ neurons across all sessions is computed using Equation 4.
Directionselective neurons
We identified DS neurons (${\text{M}}_{\text{in}}$ neurons) using the passive motionviewing task (described above). We classified a neuron as ${\text{M}}_{\text{in}}$ if it satisfies two criteria. The first criterion is a shortlatency response to the onset of random dot motion, which we defined as a fivefold increase in firing rate relative to baseline in the first 80 ms following motion onset and a greater increase in the rate of rise in activity in the same 80 ms epoch, compared to any rise in activity in the 200 ms preceding motion onset. The second criterion is direction selectivity. We calculated the area under the ROC (AUC) comparing leftward versus rightward for two separate epochs: (i) $0.15\le t<0.3$ s and (ii) $0.3\le t<0.5$ s. Neurons were determined to be DS if the AUC in either epoch exceeded 0.6. We excluded one neuron from this analysis because it switched its direction preference in the two epochs. We also excluded neurons that had previously been classified as ${\text{T}}_{\text{in}}$.
In total, 6 of the 152 ${\text{T}}_{\text{in}}^{\text{con}}$ neurons fall into this group: three per monkey; at most two in a session. Removal of these neurons has negligible effects on the findings as pairs of ${S}_{\text{Tin}}^{\text{con}}$ constructed with and without removal are strongly correlated (r = 0.9867).
Latency analysis
We estimated the latency of DS responses using the CUSUM method (Ellaway, 1978; Lorteije et al., 2015). We employed a receiver operating characteristic (ROC) analysis to estimate the selectivity of each ${\text{M}}_{\text{in}}$ neuron to motion direction. The AUC reflects the separation of the distributions of spike counts (100–400 ms after motion onset) on single trials of leftward and rightward motion, respectively. We included only correct trials with response times greater than 450 ms and motion strengths above 10% coherence. For each neuron with $\text{AUC}>0.6$, we computed the difference in spike counts (25 ms bins) between correct trials featuring leftward and rightward motion. Subsequently, we accumulated these differences over time, following the CUSUM method. The resulting difference is approximately zero before the onset of direction selectivity and then either increases or decreases monotonically, depending on the preferred motion direction. To identify the transition between these two regimes, we fit a dog leg function to the cumulative sum of spikes: a flat line starting at ${t}_{0}=0$ followed by a linearly increasing component beginning at ${t}_{1}>{t}_{0}$. The time of the end of the flat portion (between 0 and 500 ms from motion onset) of the fit was taken as the latency. Estimating latencies based on cumulative sums of spikes helps mitigate the effect of neuronal noise. The fitting step reduces the effect of the number of trials on latency estimates compared to traditional methods that rely on ttests in moving windows.
Correlations between $\mathrm{M}}_{\mathrm{i}\mathrm{n}$ and $\mathrm{T}}_{\mathrm{i}\mathrm{n}}^{\mathrm{c}\mathrm{o}\mathrm{n}$
The analysis of the correlations shown in Figure 6e is based on the spike counts of the ${\text{M}}_{\text{in}}^{\text{left}}$, ${\text{M}}_{\text{in}}^{\text{right}}$, and ${\text{T}}_{\text{in}}^{\text{con}}$ neurons calculated in 25 ms windows. We computed residuals by subtracting from each trial and time bin its average over trials of the same signed coherence. The spike count residuals were then zscored independently for each time bin and session. Trials from different sessions were concatenated, and the baseline activity—last 100 ms before motion onset—was subtracted from each trial. We refer to the resulting signals as $\stackrel{~}{S}}_{\mathrm{M}\mathrm{i}\mathrm{n}}^{\mathrm{l}\mathrm{e}\mathrm{f}\mathrm{t}$, $\stackrel{~}{S}}_{\mathrm{M}\mathrm{i}\mathrm{n}}^{\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}$, and $\stackrel{~}{S}}_{\mathrm{T}\mathrm{i}\mathrm{n}}^{\mathrm{c}\mathrm{o}\mathrm{n}$. Trials with response time less than 0.55 s were discarded, and the correlations between the difference, $\stackrel{~}{S}}_{\mathrm{M}\mathrm{i}\mathrm{n}}^{\mathrm{l}\mathrm{e}\mathrm{f}\mathrm{t}}{\stackrel{~}{S}}_{\mathrm{M}\mathrm{i}\mathrm{n}}^{\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{h}\mathrm{t}$, and $\stackrel{~}{S}}_{\mathrm{T}\mathrm{i}\mathrm{n}}^{\mathrm{c}\mathrm{o}\mathrm{n}$ ,were calculated for all pairs of time steps between 0 and 500 ms (Figure 6e). Statistical significance was assessed using permutation tests, as follows. Two regions of interest (ROIs) were defined based on the time from stimulus onset for the ${\text{M}}_{\text{in}}$ ($x$) and ${\text{T}}_{\text{in}}$ ($y$) dimensions. The first region of interest, ${\text{ROI}}_{1}$, is characterized by ${t}_{x}>100$, ${t}_{y}>200$, and ${t}_{y}>{t}_{x}$. According to our hypothesis that the ${\text{M}}_{\text{in}}$ neurons represent the momentary evidence integrated by ${\text{T}}_{\text{in}}^{\text{con}}$ neurons, we anticipated high correlations in this region. The second region of interest, ${\text{ROI}}_{2}$, is defined by ${t}_{x}>100$, ${t}_{y}>200$, and ${t}_{y}<{t}_{x}$. If contrary to our hypothesis ${\text{M}}_{\text{in}}$ and ${\text{T}}_{\text{in}}$ signals were influencing each other bidirectionally, we would expect high correlations in this region. We calculated the difference in correlations between these two groups, $\u27e8{\rho}_{\text{ROI1}}\u27e9\u27e8{\rho}_{\text{ROI2}}\u27e9$, where the expectation is over the time bins within each region of interest. This difference was compared to those obtained after randomly shuffling the order of the trials for one of the dimensions before calculating the pairwise correlations (${N}_{\text{shuffles}}=200$). We assess significance with a ztest given the mean and standard deviation of the values obtained under shuffling. The analysis was repeated with an alternative ${\text{ROI}}_{2}$ defined by ${t}_{x}<100$ and ${t}_{y}<200$, representing the times before direction selectivity is present in at least one of the two dimensions.
Correlations between $\mathrm{M}}_{\mathrm{i}\mathrm{n}$ signals and behavior
To assess the leverage of ${\textstyle \text{M}}}_{{\textstyle \text{in}}$ signals on choice and RT (Figure 6d), we performed the same logistic regression and pairwise correlation analyses as in Figure 5, substituting the ${\text{M}}_{\text{in}}^{\text{left}}$ and ${\text{M}}_{\text{in}}^{\text{right}}$ for ${S}^{x}$. The leverage on choice is not mediated by a later sample of either ${\text{M}}_{\text{in}}$ signal (${\xi}^{\text{Ch}}\le 9.6\%$; not shown), and there is negligible leverage on RT to mediate. We suspect the failure to detect leverage of ${\text{M}}_{\text{in}}$ is explained by a lack of power, owing to the focus on long RT trials, narrow sample windows (50 ms boxcar), and the small number of ${\text{M}}_{\text{in}}^{\text{left}}$ and ${\textstyle \text{M}}}_{{\textstyle \text{in}}}^{{\textstyle \text{right}}$ neurons. We support this suspicion with a simpler correlation analysis using the difference of the ${\text{M}}_{\text{in}}$ signals (standardized as in the previous paragraph):
on the interval $0.1\le t\le 0.4$ s from motion onset, on each trial, k, including trials with contraversive choices and $\text{RT}\ge 500$ ms. We calculated the Pearson correlation coefficient between ${\psi}_{k}$ and RT. Response times were zscored independently for each signed motion strength and session. We evaluated the null hypothesis that the correlation coefficient is nonnegative. The reported pvalue is based on a onetailed tstatistic.
Variance and autocorrelation of smoothed diffusion signals
The analyses in Figure 3 compare the variance and autocorrelation of the singletrial signals, ${S}^{\mathrm{ramp}}(t)$, to those expected from unbounded driftdiffusion. To mitigate the effect of the bound, we focus on the earliest epoch of putative integration (200–506 ms after motion onset; six 51 ms counting windows) and the weakest motion strengths ($\text{coh}\le 3.2\%)$. The singletrial signals are detrended by the mean across trials sharing the same signed motion coherence and baseline corrected by subtraction of ${S}_{i}^{\mathrm{ramp}}(t=0.2)$ from all time points on each trial $i$.
The variance as a function of time and the autocorrelation as a function of time and lag are well specified for the cumulative sum of discrete iid random samples, but the autocorrelation is affected by the boxcar filter we applied to render the signals. We incorporated the correction in our characterization of unbounded diffusion. The derivation is summarized in Appendix 1, and we provide MATLAB code in the GitHub repository. The theoretical values shown in Figure 3 assume a 1 kHz sampling rate and standard Wiener process (i.e., samples drawn from a normal distribution with $\{\mu =0,\sigma =\sqrt{dt}\}$). The evolution of variance would be a line from 0 to 1 over the first second of integration. The key prediction, shown in Figure 3a, is that the variance of mean singletrial signals, ${S}_{i}(t)$, over the epoch $26\pm \mathrm{\hspace{0.17em}25}$ ms should double in the epoch $(26+51)\pm 25$ ms, and triple in the epoch ($26+2\times 51)\pm 25$ ms), and so on for each successive nonoverlapping running mean. We therefore use arbitrary units, normalized to the measured variance of the first point. We do not know the variance of the driftdiffusion signal that $S(t)$ is thought to approximate, but we assume it can be decomposed—by the law of total variance—to a component given by driftdiffusion and components associated with spiking and other nuisance factors. We therefore search for a scalar nonnegative factor $\varphi \le 1$ that multiplies all terms in the diagonal of the empirical covariance matrix (i.e., the variance) before normalizing to produce the autocorrelation matrix. We search for the value of $\varphi $ that minimizes the sum of squares between Fisherz transformed correlation coefficients in the theoretical and empirical autocorrelation matrices (Figure 3b and c). Standard errors of the variance and autocorrelations in Figure 3a and c are estimated by a bootstrap procedure respecting the composition of motion strength and direction (the s.e. is standard deviation of each variance and autocorrelation term across 500 repetitions of the procedure.
Appendix 1
We consider a discrete time (sampling interval $dt$) Wiener process with independent random increments ${\u03f5}_{k}$ on time step $k$ that are zeromean noise with variance ${\sigma}^{2}({\u03f5}_{k})=dt$ (i.e., unit variance per second). The accumulated evidence (i.e., decision variable, DV) on time step $p$ is
For such a Wiener process,
As the increments ${\u03f5}_{k}$ are independent across time, ${\textstyle \text{Cov}}({\u03f5}_{j},{\u03f5}_{k})$ is 0 for $j\ne k$ and $dt$ for $j=k$
We define the mean DV over a window of $\pm n$ points as
We consider two time points $p<q$ with window $n$ such that there is no overlap and hence $p+n<qn$
Given that $\sum _{j=1}^{n}\sum _{k=1}^{n}min(j,k)=\sum _{j=1}^{n}{j}^{2}=n(n+1)(2n+1)/6$
Therefore, the correlation
In contrast for the point estimates at $p$ and $q$
It is useful to reexpress the above two equations in terms of actual time ${t}_{p}$ and ${t}_{q}$ and window size ${t}_{n}$. Substituting for $p$, $q$, and $n$ with ${t}_{p}/dt$, ${t}_{q}/dt$ and ${t}_{n}/dt$
Data availability
Matlab code for all analyses and graphs are available at GitHub (copy archived at Steinemann, 2024). The data are deposited at Zenodo.

ZenodoData for "Direct observation of the neural computations underlying a single decision".https://doi.org/10.5281/zenodo.13207505
References

BookThe neuron doctrine in perceptionIn: Gazzaniga M, editors. The Cognitive Neurosciences, The Cognitive Neurosciences. Boston: MIT Press. pp. 415–435.

Normalization as a canonical neural computationNature Reviews. Neuroscience 13:51–62.https://doi.org/10.1038/nrn3136

Decisionmaking with multiple alternativesNature Neuroscience 11:693–702.https://doi.org/10.1038/nn.2123

Decisions in changing conditions: the urgencygating modelThe Journal of Neuroscience 29:11560–11571.https://doi.org/10.1523/JNEUROSCI.184409.2009

Representation of accumulating evidence for a decision in two parietal areasThe Journal of Neuroscience 35:4306–4318.https://doi.org/10.1523/JNEUROSCI.245114.2015

Microstimulation of visual cortex affects the speed of perceptual decisionsNature Neuroscience 6:891–898.https://doi.org/10.1038/nn1094

The cost of accumulating evidence in perceptual decision makingThe Journal of Neuroscience 32:3612–3628.https://doi.org/10.1523/JNEUROSCI.401011.2012

Cumulative sum technique and its application to the analysis of peristimulus time histogramsElectroencephalography and Clinical Neurophysiology 45:302–304.https://doi.org/10.1016/00134694(78)900172

Direction selectivity of neurons in the macaque lateral intraparietal areaJournal of Neurophysiology 101:289–305.https://doi.org/10.1152/jn.00400.2007

Comparison of corticocortical and corticocollicular signals for the generation of saccadic eye movementsJournal of Neurophysiology 87:845–858.https://doi.org/10.1152/jn.00317.2001

A proposed common neural mechanism for categorization and perceptual decisionsNature Neuroscience 14:143–146.https://doi.org/10.1038/nn.2740

Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortexJournal of Neurophysiology 61:331–349.https://doi.org/10.1152/jn.1989.61.2.331

Memory related motor planning activity in posterior parietal cortex of macaqueExperimental Brain Research 70:216–220.https://doi.org/10.1007/BF00271862

The neural basis of decision makingAnnual Review of Neuroscience 30:535–574.https://doi.org/10.1146/annurev.neuro.29.051605.113038

Elapsed decision time affects the weighting of prior probability in a perceptual decision taskThe Journal of Neuroscience 31:6339–6352.https://doi.org/10.1523/JNEUROSCI.561310.2011

BookUnixBased MultipleProcess System, for RealTime Data Acquisition and ControlOSTI.GOV.

Visual and oculomotor functions of monkey substantia nigra pars reticulataIII. MemoryContingent Visual and Saccade Responses. Journal of Neurophysiology 49:1268–1284.https://doi.org/10.1152/jn.1983.49.5.1268

Characterizing the dynamics of mental representations: the temporal generalization methodTrends in Cognitive Sciences 18:203–210.https://doi.org/10.1016/j.tics.2014.01.002

A role for neural integrators in perceptual decision makingCerebral Cortex 13:1257–1269.https://doi.org/10.1093/cercor/bhg097

Stimuli reduce the dimensionality of cortical activityFrontiers in Systems Neuroscience 10:11.https://doi.org/10.3389/fnsys.2016.00011

Signal multiplexing and singleneuron computations in lateral intraparietal area during decisionmakingThe Journal of Neuroscience 33:2254–2267.https://doi.org/10.1523/JNEUROSCI.298412.2013

Monkey posterior parietal cortex neurons antidromically activated from superior colliculusJournal of Neurophysiology 78:3493–3497.https://doi.org/10.1152/jn.1997.78.6.3493

Modeling response times for twochoice decisionsPsychological Science 9:347–356.https://doi.org/10.1111/14679280.00067

Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time taskThe Journal of Neuroscience 22:9475–9489.https://doi.org/10.1523/JNEUROSCI.222109475.2002

Microstimulation in visual area MT: effects on direction discrimination performanceThe Journal of Neuroscience 12:2331–2355.https://doi.org/10.1523/JNEUROSCI.120602331.1992

Neural basis of saccade target selectionReviews in the Neurosciences 6:63–85.https://doi.org/10.1515/revneuro.1995.6.1.63

A computational analysis of the relationship between neuronal and behavioral responses to visual motionThe Journal of Neuroscience 16:1486–1510.https://doi.org/10.1523/JNEUROSCI.160401486.1996

Comparison of decisionrelated signals in sensory and motor preparatory responses of neurons in area LIPThe Journal of Neuroscience 38:6350–6365.https://doi.org/10.1523/JNEUROSCI.066818.2018

Topographic maps in human frontal and parietal cortexTrends in Cognitive Sciences 13:488–495.https://doi.org/10.1016/j.tics.2009.08.005

SoftwareLIP_single_trial_decision_variable, version swh:1:rev:869a42420134af6113d3f2e0f27b090078153185Software Heritage.

Computation through neural population dynamicsAnnual Review of Neuroscience 43:249–275.https://doi.org/10.1146/annurevneuro092619094115

Evidence accumulation occurs locally in the parietal cortexNature Communications 13:4426.https://doi.org/10.1038/s41467022322106
Article and author information
Author details
Funding
Howard Hughes Medical Institute
 Ariel Zylberberg
 Michael N Shadlen
National Institutes of Health BRAIN Initiative (R01NS113113)
 Natalie Steinemann
 Michael N Shadlen
National Eye Institute (T32 EY013933)
 Gabriel M Stine
Grossman Center (Zuckerman Institute)
 Eric Trautmann
Brain and Behavior Research Foundation
 Eric Trautmann
National Eye Institute (F31 EY032791)
 Gabriel M Stine
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Acknowledgements
We thank Shushruth, NaYoung So, and David Gruskin for comments on the manuscript, Cornel Duhaney and Brian Madeira for their assistance in the planning and execution of surgeries, animal training and general support, and we thank Columbia University’s ICM for the quality of care they provide for our animals, especially during the pandemic and lockdown. We would further like to thank Tanya Tabachnik and her team at the Zuckerman Institute Advanced Instrumentation Core and Tim Harris, Weilung Sun, Jennifer Colonell, and Bill Karsh at HHMI Janelia for their continued support with Neuropixels1.0NHP45 probes development and testing. This research was supported by the Howard Hughes Medical Institute; an R01 grant from the NIH Brain Initiative (MNS, R01NS113113); a T32 and F31 grant from the National Eye Institute (GMS, T32 EY013933, F31 EY032791); the Grossman center; and the Brain and Behavior Research Foundation. DMW is a consultant to CTRLLabs Inc, in the Reality Labs Division of Meta. This entity did not support or influence this work.
Ethics
All training, surgery, and experimental procedures complied with guidelines from the National Institutes of Health and were approved by the Institutional Animal Care and Use Committee at Columbia University (protocols AAAN4900 and ACAAAW4454).
Version history
 Sent for peer review:
 Preprint posted:
 Reviewed Preprint version 1:
 Reviewed Preprint version 2:
 Version of Record published:
Cite all versions
You can cite all versions using the DOI https://doi.org/10.7554/eLife.90859. This DOI represents all versions, and will always resolve to the latest one.
Copyright
© 2023, Steinemann, Stine et al.
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics

 4,116
 views

 267
 downloads

 5
 citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading

 Neuroscience
Background:
Poststroke epilepsy (PSE) is a critical complication that worsens both prognosis and quality of life in patients with ischemic stroke. An interpretable machine learning model was developed to predict PSE using medical records from four hospitals in Chongqing.
Methods:
Medical records, imaging reports, and laboratory test results from 21,459 ischemic stroke patients were collected and analyzed. Univariable and multivariable statistical analyses identified key predictive factors. The dataset was split into a 70% training set and a 30% testing set. To address the class imbalance, the Synthetic Minority Oversampling Technique combined with Edited Nearest Neighbors was employed. Nine widely used machine learning algorithms were evaluated using relevant prediction metrics, with SHAP (SHapley Additive exPlanations) used to interpret the model and assess the contributions of different features.
Results:
Regression analyses revealed that complications such as hydrocephalus, cerebral hernia, and deep vein thrombosis, as well as specific brain regions (frontal, parietal, and temporal lobes), significantly contributed to PSE. Factors such as age, gender, NIH Stroke Scale (NIHSS) scores, and laboratory results like WBC count and Ddimer levels were associated with increased PSE risk. Treebased methods like Random Forest, XGBoost, and LightGBM showed strong predictive performance, achieving an AUC of 0.99.
Conclusions:
The model accurately predicts PSE risk, with treebased models demonstrating superior performance. NIHSS score, WBC count, and Ddimer were identified as the most crucial predictors.
Funding:
The research is funded by Central University basic research young teachers and students research ability promotion subprojec t(2023CDJYGRHZD06), and by Emergency Medicine Chongqing Key Laboratory Talent Innovation and development joint fund project (2024RCCX10).

 Neuroscience
The entorhinal cortex (EC) connects to the hippocampus sending different information from cortical areas that is first processed at the dentate gyrus (DG) including spatial, limbic and sensory information. Excitatory afferents from lateral (LPP) and medial (MPP) perforant pathways of the EC connecting to granule cells of the DG play a role in memory encoding and information processing and are deeply affected in humans suffering Alzheimer’s disease and temporal lobe epilepsy, contributing to the dysfunctions found in these pathologies. The plasticity of these synapses is not well known yet, as are not known the forms of longterm depression (LTD) existing at those connections. We investigated whether spike timingdependent longterm depression (tLTD) exists at these two different ECDG synaptic connections in mice, and whether they have different action mechanisms. We have found two different forms of tLTD, at LPP and MPPGC synapses and characterised their cellular and intracellular mechanistic requirements. We found that both forms of tLTD are expressed presynaptically and that whereas tLTD at LPPGC synapses does not require NMDAR, tLTD at MPPGC synapses requires ionotropic NMDAR containing GluN2A subunits. The two forms of tLTD require different group I mGluR, mGluR5 LPPGC synapses and mGluR1 MPPGC synapses. In addition, both forms of tLTD require postsynaptic calcium, eCB synthesis, CB_{1}R, astrocyte activity, and glutamate released by astrocytes. Thus, we discovered two novel forms of tLTD that require astrocytes at ECGC synapses.