How systems solve complex cognitive tasks is a fundamental question in neuroscience and artificial intelligence (Rigotti et al., 2013; Yang et al., 2019; Wang et al., 2018; Silver et al., 2016; Jensen et al., 2023). A key aspect of complex tasks is that they often involve multiple types of stimuli, some of which can even be irrelevant for performing the correct behavioral response (Freedman et al., 2001; Mante et al., 2013; Parthasarathy et al., 2017) or predicting reward (Bernardi et al., 2020; Chadwick et al., 2023). Over the course of task exposure, subjects must typically identify which stimuli are relevant and which are irrelevant. Examples of irrelevant stimuli include those that are irrelevant at all times in a task (Freedman et al., 2001; Chadwick et al., 2023; Duncan, 2001; Rainer et al., 1998; Stokes et al., 2013) – which we refer to as static irrelevance – and stimuli that are relevant at some time points but are irrelevant at other times in a trial – which we refer to as dynamic irrelevance (e.g. as is often the case for context-dependent decision-making tasks Mante et al., 2013; Flesch et al., 2022; Monsell, 2003; Braver, 2012). Although tasks involving irrelevant stimuli have been widely used, it remains an open question as to how different types of irrelevant stimuli should be represented, in combination with relevant stimuli, to enable optimal task performance.

One may naively think that statically irrelevant stimuli should always be suppressed. However, stimuli that are currently irrelevant may be relevant in a future task. Furthermore, it is unclear whether dynamically irrelevant stimuli should be suppressed at all since the information is ultimately needed by the circuit. It may therefore be beneficial for a neural circuit to represent irrelevant information as long as no unnecessary costs are incurred and task performance remains high. Several factors could have a strong impact on whether irrelevant stimuli affect task performance. For example, levels of neural noise in the circuit as well as energy constraints and the metabolic costs of overall neural activity (Flesch et al., 2022; Whittington et al., 2022; Sussillo et al., 2015; Orhan and Ma, 2019; Löwe, 2023; Cueva and Wei, 2018; Luo et al., 2023; Kao et al., 2021; Deneve et al., 2001; Barak et al., 2013) can affect how stimuli are represented in a neural circuit. Indeed, both noise and metabolic costs are factors that biological circuits must contend with (Tomko and Crapper, 1974; Laughlin, 2001; Churchland et al., 2006; Hasenstaub et al., 2010). Despite these considerations, models of neural population codes, including hand-crafted models and optimized artificial neural networks, typically use only a very limited range of the values of such factors (Wang et al., 2018; Mante et al., 2013; Sussillo et al., 2015; Barak et al., 2013; Cueva et al., 2020; Driscoll et al., 2022; Song et al., 2016; Echeveste et al., 2020; Stroud et al., 2021) (but also see Yang et al., 2019; Orhan and Ma, 2019). Therefore, despite the success of recent comparisons between neural network models and experimental recordings (Wang et al., 2018; Mante et al., 2013; Sussillo et al., 2015; Cueva and Wei, 2018; Barak et al., 2013; Cueva et al., 2020; Echeveste et al., 2020; Stroud et al., 2021; Lindsay, 2021), we may only be recovering very few out of a potentially large range of different representational strategies that neural networks could exhibit (Schaeffer et al., 2022).

One challenge for distinguishing between different representational strategies, particularly when analyzing experimental recordings, is that some stimuli may simply be represented more strongly than others. In particular, we might expect stimuli to be strongly represented in cortex a priori if they have previously been important to the animal. Indeed, being able to represent a given stimulus when learning a new task is likely a prerequisite for learning whether it is relevant or irrelevant in that particular context (Rigotti et al., 2013; Bernardi et al., 2020). Previously, it has been difficult to distinguish between whether a given representation existed a priori or emerged as a consequence of learning because neural activity is typically only recorded after a task has already been learned. A more relevant question is how the representation changes over learning (Chadwick et al., 2023; Reinert et al., 2021; Durstewitz et al., 2010; Schuessler et al., 2020; Costa et al., 2017), which provides insights into how the specific task of interest affects the representational strategy used by an animal or artificial network (Richards et al., 2019).

To resolve these questions, we optimized recurrent neural networks on a task that involved two types of irrelevant stimuli. One feature of the stimulus was statically irrelevant, and another feature of the stimulus was dynamically irrelevant. We found that, depending on the neural noise level and metabolic cost that was imposed on the networks during training, a range of representational strategies emerged in the optimized networks, from maximal (representing all stimuli) to minimal (representing only relevant stimuli). We then compared the strategies of our optimized networks with learning-resolved recordings from the prefrontal cortex (PFC) of monkeys exposed to the same task. We found that the representational geometry of the neural recordings changed in line with the minimal strategy. Using a simplified model, we derived mathematically how the strength of relevant and irrelevant coding depends on the noise level and metabolic cost. We then confirmed our theoretical predictions in both our task-optimized networks and neural recordings. By reverse-engineering our task-optimized networks, we also found that activity-silent, sub-threshold dynamics led to the suppression of dynamically irrelevant stimuli, and we confirmed predictions of this mechanism in our neural recordings.

In summary, we provide a mechanistic understanding of how different representational strategies can emerge in both biological and artificial neural circuits over the course of learning in response to salient biological factors such as noise and metabolic costs. These results in turn explain why PFC appears to employ a minimal representational strategy by filtering out task-irrelevant information.

Comparing the neural representations of task-optimized networks with those observed in experimental data has been particularly fruitful in recent years (Wang et al., 2018; Mante et al., 2013; Sussillo et al., 2015; Barak et al., 2013; Cueva et al., 2020; Echeveste et al., 2020; Stroud et al., 2021; Lindsay, 2021). However, networks are typically optimized using only a very limited range of hyperparameter values (Wang et al., 2018; Mante et al., 2013; Sussillo et al., 2015; Barak et al., 2013; Cueva et al., 2020; Driscoll et al., 2022; Song et al., 2016; Echeveste et al., 2020; Stroud et al., 2021). Instead, here we showed that different settings of key, biologically relevant hyperparameters such as noise and metabolic costs, can yield a variety of qualitatively different dynamical regimes that bear varying degrees of similarity with experimental data. In general, we found that increasing levels of noise and firing rate regularization led to increasing amounts of irrelevant information being filtered out in the networks. Indeed, filtering out of task-irrelevant information is a well-known property of the PFC and has been observed in a variety of tasks (Freedman et al., 2001; Duncan, 2001; Cueva et al., 2020; Reinert et al., 2021; Miller and Cohen, 2001; Rainer and Miller, 2002; Asaad et al., 2000; Everling et al., 2002). We provide a mechanistic understanding of the specific conditions that lead to stronger filtering of task-irrelevant information. We predict that these results should also generalize to richer, more complex cognitive tasks that may, for example, require context switching (Flesch et al., 2022; Reinert et al., 2021; Asaad et al., 2000) or invoke working memory (Freedman et al., 2001; Rainer et al., 1998; Asaad et al., 2000). Indeed, filtering out of task-irrelevant information in the PFC has been observed in such tasks (Freedman et al., 2001; Rainer et al., 1998; Flesch et al., 2022; Reinert et al., 2021; Asaad et al., 2000; Mack et al., 2020).

Our results are also likely a more general finding of neural circuits that extend beyond the PFC. In line with this, it has previously been shown that strongly regularized neural network models trained to reproduce monkey muscle activities during reaching bore a stronger resemblance to neural recordings from primary motor cortex compared to unregularized models (Sussillo et al., 2015). In related work on motor control, recurrent networks controlled by an optimal feedback controller recapitulated key aspects of experimental recordings from primary motor cortex (such as orthogonality between preparatory and movement neural activities) when the control input was regularized (Kao et al., 2021). Additionally, regularization of neural firing rates, and its natural biological interpretation as a metabolic cost, has recently been shown to be a key ingredient for the formation of grid cell-like response profiles in artificial networks (Whittington et al., 2022; Cueva and Wei, 2018).

By showing that PFC representations changed in line with a minimal representational strategy, our results are in line with various studies showing low-dimensional representations under a variety of tasks in the PFC and other brain regions (Rainer et al., 1998; Flesch et al., 2022; Cueva et al., 2020; Ganguli et al., 2008; Sohn et al., 2019). This is in contrast to several previous observations of high-dimensional neural activity in PFC (Rigotti et al., 2013; Bernardi et al., 2020). Both high- and low-dimensional regimes confer distinct yet useful benefits: high-dimensional representations allow many behavioral readouts to be generated, thereby enabling highly flexible behavior (Rigotti et al., 2013; Flesch et al., 2022; Barak et al., 2013; Enel et al., 2016; Maass et al., 2002; Fusi et al., 2016), whereas low-dimensional representations are more robust to noise and allow for generalization across different stimuli (Flesch et al., 2022; Barak et al., 2013; Fusi et al., 2016). These two different representational strategies have previously been studied in models by setting the initial network weights to either small values (to generate low-dimensional ‘rich’ representations) or large values (Flesch et al., 2022) (to generate high-dimensional ‘lazy’ representations). However, in contrast to previous approaches, we studied the more biologically plausible effects of firing rate regularization (i.e. a metabolic cost; see also the supplement of Flesch et al., 2022) on the network activities over the course of learning and compared them to learning-related changes in PFC neural activities. Firing rate regularization will cause neural activities to wax and wane as networks are exposed to new tasks depending upon the stimuli that are currently relevant. In line with this, it is conceivable that prolonged exposure to a task that has a limited number of stimulus conditions, some of which can even be generalized over (as was the case for our task), encourages more low-dimensional dynamics to form (Wójcik, 2023; Flesch et al., 2022; Dubreuil et al., 2022; Fusi et al., 2016; Musslick, 2017; Mastrogiuseppe and Ostojic, 2018). In contrast, tasks that use a rich variety of stimuli (that may even dynamically change over the task Wang et al., 2018; Jensen et al., 2023; Heald et al., 2021), and which do not involve generalization across stimulus conditions, may more naturally lead to high-dimensional representations (Rigotti et al., 2013; Barak et al., 2013; Dubreuil et al., 2022; Fusi et al., 2016; Mastrogiuseppe and Ostojic, 2018; Bartolo et al., 2020). It would therefore be an important future question to understand how our results also depend on the task being studied as some tasks may more naturally lead to the ‘maximal’ representational regime (Barak et al., 2013; Dubreuil et al., 2022; Mastrogiuseppe and Ostojic, 2018; Figures 1—3 and 5, blue shading).

A previous analysis of the same dataset that we studied here focused on the late parts of the trial (Wójcik, 2023). In particular, they found that the final result of the computation needed for the task, the XOR operation between color and shape, emerges and eventually comes to dominate lPFC representations over the course of learning in the late shape period (Wójcik, 2023). Our analysis goes beyond this by studying mechanisms of suppression of both static and dynamically irrelevant stimuli across all task periods and how different levels of neural noise and metabolic cost can lead to qualitatively different representations of irrelevant stimuli in task-optimized recurrent networks. Other previous studies focused on characterizing the representation of several task-relevant (Rigotti et al., 2013; Bernardi et al., 2020; Stokes et al., 2013) – and, in some cases, -irrelevant (Flesch et al., 2022) – variables over the course of individual trials. Characterizing how key aspects of neural representations change over the course of learning, as we did here, offers unique opportunities for studying the functional objectives shaping neural representations (Richards et al., 2019).

There were several aspects of the data that were not well captured by our models. For example, during the shape period, decodability of shape decreased while decodability of color increased (although not significantly) in our neural recordings (Figure 3a). These differences in changes in decoding may be due to fundamentally different ways that brains encode sensory information upstream of PFC, compared to our models. For example, shape and width are both geometric features of the stimulus, whose encoding is differentiated from that of color already at early stages of visual processing (Kandel, 2000). Such a hierarchical representation of inputs may automatically lead to the (un)learning about the relevance of width (which the ${\displaystyle {\sigma }_m,{\lambda }_m}$ model reproduced) generalizing to shape, but not to color. In contrast, inputs in our model used a non-hierarchical one-hot encoding (Figure 2), which did not allow for such generalization. Moreover, in the data, we may expect width to be a priori more strongly represented than color or shape because it is a much more potent sensory feature. In line with this, in our neural recordings, we found that width was very strongly represented in early learning compared to the other stimulus features (Figure 3a, far right) and width always yielded high cross-generalized decoding – even after learning (Figure 3—figure supplement 1a, far right). Nevertheless, studying changes in decoding over learning, rather than absolute decoding levels, allowed us to focus on features of learning that do not depend on the details of upstream sensory representations of stimuli. Future studies could incorporate aspects of sensory representations that we ignored here by using stimulus inputs with which the model more faithfully reproduces the experimentally observed initial decodability of stimulus features.

In line with previous studies (Yang et al., 2019; Whittington et al., 2022; Sussillo et al., 2015; Orhan and Ma, 2019; Kao et al., 2021; Cueva et al., 2020; Driscoll et al., 2022; Song et al., 2016; Stroud et al., 2021; Masse et al., 2019; Schimel et al., 2023), we operationalized metabolic cost in our models through L₂ firing rate regularization. This cost penalizes high overall firing rates. There are however alternative conceivable ways to operationalize a metabolic cost; e.g., L₁ firing rate regularization has been used previously when optimizing neural networks and promotes more sparse neural firing (Yang et al., 2019). Interestingly, although our L₂ is generally conceived to be weaker than L₁ regularization, we still found that it encouraged the network to use purely sub-threshold activity in our task. The regularization of synaptic weights may also be biologically relevant (Yang et al., 2019) because synaptic transmission uses the most energy in the brain compared to other processes (Faria-Pereira and Morais, 2022; Harris et al., 2012). Additionally, even sub-threshold activity could be regularized as it also consumes energy (although orders of magnitude less than spiking Zhu et al., 2019). Therefore, future work will be needed to examine how different metabolic costs affect the dynamics of task-optimized networks.

We build on several previous studies that have also analyzed learning-related changes in PFC activity (Wójcik, 2023; Reinert et al., 2021; Durstewitz et al., 2010; Bartolo et al., 2020) – although these studies typically used reversal-learning paradigms in which animals are already highly task proficient and the effects of learning and task switching are inseparable. For example, in a rule-based categorization task in which the categorization rule changed after learning an initial rule, neurons in mouse PFC adjusted their selectivity depending on the rule such that currently irrelevant information was not represented (Reinert et al., 2021). Similarly, neurons in rat PFC transition rapidly from representing a familiar rule to representing a completely novel rule through trial-and-error learning (Durstewitz et al., 2010). Additionally, the dimensionality of PFC representations was found to increase as monkeys learned the value of novel stimuli (Bartolo et al., 2020). Importantly, however, PFC representations did not distinguish between novel stimuli when they were first chosen. It was only once the value of the stimuli were learned that their representations in PFC were distinguishable (Bartolo et al., 2020). These results are consistent with our results where we found poor XOR decoding during the early stages of learning which then increased over learning as the monkeys learned the rewarded conditions (Figure 3a, far left). However, we also observed high decoding of width during early learning which was not predictive of reward (Figure 3a, far right). One key distinction between our study and that of Bartolo et al., 2020, is that our recordings commenced from the first trial the monkeys were exposed to the task. In contrast, in Bartolo et al., 2020, the monkeys were already highly proficient at the task and so the neural representation of their task was already likely strongly task specific by the time recordings were taken.

In line with our approach here, several recent studies have also examined the effects of different hyperparameter settings on the solution that optimized networks exhibit. One study found that decreasing regularization on network weights led to more sequential dynamics in networks optimized to perform working memory tasks (Orhan and Ma, 2019). Another study found that the number of functional clusters that a network exhibits does not depend strongly on the strength of (L₁ rate or weight) regularization, but did depend upon whether the single neuron nonlinearity saturates at high firing rates (Yang et al., 2019). It has also been shown that networks optimized to perform path integration can exhibit a range of different properties, from grid cell-like receptive fields to distinctly non grid cell-like receptive fields, depending upon biologically relevant hyperparameters – including noise and regularization (Whittington et al., 2022; Cueva and Wei, 2018; Schaeffer et al., 2022). Indeed, in addition to noise and regularization, various other hyperparameters have also been shown to affect the representational strategy used by a circuit, such as the firing rate nonlinearity (Yang et al., 2019; Whittington et al., 2022; Schaeffer et al., 2022) and network weight initialization (Flesch et al., 2022; Schaeffer et al., 2022). It is therefore becoming increasingly clear that analyzing the interplay between learning and biological constraints will be key for understanding the computations that various brain regions perform.

In this supplementary appendix, we derive the optimal linear decoder for a linear network that receives both relevant and irrelevant inputs (Problem definition and The optimal linear decoder). We derive how both the performance of the optimal decoder (The performance of the optimal linear decoder) and the metabolic cost of the network (Metabolic cost) depend on noise and the strength of relevant and irrelevant inputs. We also derive how the optimal setting of the relevant and irrelevant inputs, when jointly optimizing for both performance and a metabolic cost, depend on noise and the strength of metabolic cost (Qualitative predictions about optimal parameters). Finally, we also study how noise and the strength of metabolic cost affect the curvature of the loss function around the optimum (Curvature of the loss function around the optimum).

Mathematical analysis of relevant and irrelevant stimulus coding in a linear network

Even though in the main text we simulate and numerically analyze neural networks with nonlinear temporal dynamics that solve an XOR task, for analytical tractability, our mathematical analyses below are for a model in which responses to different inputs combine linearly. Our analysis is agnostic as to whether responses come about by simple, instantaneous feedforward or temporally extended recurrent interactions, and is only concerned with the phenomenological mapping between stimuli and the resulting response distributions. As such, we cannot distinguish between dynamically and statically irrelevant stimuli, and so we only include a single relevant and single (statically) irrelevant stimulus in our analysis. Nevertheless, as we show in the main text (Figure 4), some of the key insights from our analysis of such simplified systems generalize to the original setting.

Problem definition

We consider the responses, ${\displaystyle \mathbf{r}}$, of a neural population of $N$ neurons to a stimulus that has a ‘relevant’ feature, c, which we assume to be binary with values c₁ and c₂ (occurring with uniform frequency), and to a (scalar) irrelevant feature, $ϵ$ that is continuous (for convenience). We further assume that the responses are determined by a linear interaction of the relevant and irrelevant features and also subject to unspecific neural noise:

(S1) $\mathbf{r}=\mathit{\mu }+{\delta }_{c,c_1}\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}}{2}-{\delta }_{c,c_2}\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}}{2}+ϵ\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}{2}+\mathit{\eta }$

where µ is the grand average response, averaging across both the relevant and the irrelevant feature, ${\displaystyle \mathrm{\Delta }{\mathbf{r}}_{\text{rel}}}$ is the relevant tuning of the population, ${\displaystyle \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}$ is the irrelevant tuning, ${\displaystyle \mathit{ϵ}}$ is (without loss of generality) zero mean and unit variance variability in the irrelevant feature, and ${\displaystyle \mathit{\eta }\sim \mathcal{N}\left(0,{\sigma }^2\mathit{I}\right)}$ is other neural noise.

In this note, we study how this neural population can optimize the decodability of c while also balancing metabolic cost (defined below). Specifically, we will regard µ and ${\sigma }^2$ as givens (constraints; one could also consider µ as an optimizable parameter, and its optimal value would simply be 0) and ask how the population should ‘choose’ ${\displaystyle \mathrm{\Delta }{\mathbf{r}}_{\text{rel}}}$ and ${\displaystyle \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}$ for this trade-off (or some summary statistics thereof, see below).

The optimal linear decoder

We first study in general how information can be decoded from the population. The statistically optimal decoder, decoding c from ${\displaystyle \mathbf{r}}$, is the maximum likelihood decoder (note that we have assumed c₁ and c₂ to occur with equal probabilities, see above). To make the derivations tractable, from here on we assume that $ϵ$ is specifically Gaussian distributed, i.e., ${\displaystyle ϵ\sim \mathcal{N}\left(0,1\right)}$, unless otherwise noted. In this case, the distribution of responses conditioned on the relevant stimulus feature becomes (equivariate) Gaussian, with some effective noise covariance $\mathit{\Sigma }$ (to be determined later, Equation S13):

(S2) $\mathit{r}|c_1\sim \mathcal{N}\left(\mathit{\mu }+\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}}{2},\mathbf{\Sigma }\right)$

(S3) $\mathit{r}|c_2\sim \mathcal{N}\left(\mathit{\mu }-\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}}{2},\mathbf{\Sigma }\right)$

Thus, the log odds ratio is

(S4) $z=\mathit{l}\mathit{n}\frac{\mathcal{P}\left(\mathbf{r}|c_1\right)}{\mathcal{P}\left(\mathbf{r}|c_2\right)}=\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathbf{r}-\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathit{\mu }$

and the optimal decoder responds ‘1’ if

(S5) $z>0$

Thus, in this case, the optimal decoder is linear in ${\displaystyle \mathbf{r}}$, or conversely, a linear decoder (with the correct coefficients) is optimal.

The performance of the optimal linear decoder

We now turn to the question of how the parameters of the neural population affect its decodability, i.e., the performance of the optimal linear decoder. (We still assume that $ϵ$, and thus ${\displaystyle \mathbf{r}|c_{1/2}}$, is Gaussian distributed.)

As we saw, the optimal decoder can be described by a simple thresholding of the log odds (Equation S5). The log odds, z, itself is also Gaussian distributed conditioned on the relevant feature, because ${\displaystyle \mathbf{r}}$ is Gaussian distributed (Equations S2 and S3) and z is a linear function of it (Equation S4):

(S6) $z|c_1\sim \mathcal{N}\left({\mu }_z,{\sigma }_z^2\right)$

and, due to symmetry,

(S7) $z|c_2\sim \mathcal{N}\left(-{\mu }_z,{\sigma }_z^2\right)$

with

(S8) ${\mu }_z=\mathbb{E}\left[z|c_1\right]=-\mathbb{E}\left[z|c_2\right]=\frac{1}{2}\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}$

and

(S9) ${\sigma }_z^2=\mathbb{V}\left[z|c_1\right]=\mathbb{V}\left[z|c_2\right]=\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}{\mathbf{\Sigma }}^{-1}\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}=2{\mu }_z$

The probability of correct decoding for c₁ (and by symmetry, also for c₂, and thus also after averaging over c) is given by

(S10) $\mathrm{\Pi }=\mathcal{P}\left(\text{respond}“1”|c_1\right)=\mathcal{P}\left(\text{respond}“2”|c_2\right)$

(S11) $={\int }_0^{\mathrm{\infty }}\mathcal{N}\left(z;{\mu }_z,{\sigma }_z^2\right)dz=\mathrm{\Psi }\left(\frac{{\mu }_z}{{\sigma }_z}\right)=\mathrm{\Psi }\left(\sqrt{\frac{{\mu }_z}{2}}\right)$

Therefore, the performance of the optimal linear decoder scales monotonically with μ_z. To gain further insight into what this means in our particular setting, let us now express the effective noise covariance matrix ${\displaystyle \mathbf{\Sigma }}$ with the parameters of the problem definition:

(S12) $\mathbf{\Sigma }=\mathbb{C}\left[\mathbf{r}|c_1\right]=\mathbb{C}\left[\mathbf{r}|c_2\right]$

(S13) $={\sigma }^2\mathbf{I}+\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}^{\mathrm{\top }}}{4}$

and its inverse (expressed using the Sherman–Morrison formula):

(S14) ${\mathbf{\Sigma }}^{-1}=\frac{1}{{\sigma }^2}\left[\mathbf{I}-\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}^{\mathrm{\top }}}{4{\sigma }^2+\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}\right]$

Substituting Equation S14 into the formula for μ_z (Equation S8), we obtain:

(S15) ${\mu }_z=\frac{1}{2{\sigma }^2}\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}-\frac{1}{2{\sigma }^2}\frac{\left(\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}\right)\left(\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}\right)}{4{\sigma }^2+\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}$

(S16) $=\frac{1}{2{\sigma }^2}{\Vert \mathrm{\Delta }{\mathbf{r}}_{\text{rel}}\Vert }^2\left[1-\frac{{\Vert \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}\Vert }^2}{4{\sigma }^2+{\Vert \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}\Vert }^2}{\left(\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}{\Vert \mathrm{\Delta }{\mathbf{r}}_{\text{rel}}\Vert \Vert \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}\Vert }\right)}^2\right]$

Thus, μ_z can be simply expressed as

(S17) ${\mu }_z=\frac{1}{2{\sigma }^2}{\gamma }^2\left(1-\frac{{\alpha }^2}{4{\sigma }^2+{\alpha }^2}{\beta }^2\right)$

where

(S18) $\gamma =\Vert \mathrm{\Delta }{\mathbf{r}}_{\text{rel}}\Vert \text{is the magnitude of tuning to the relevant feature}$

(S19) $\alpha =\Vert \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}\Vert \text{is the magnitude of tuning to the irrelevant feature}$

(S20) $\text{and}\beta =\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}{\gamma \alpha }\text{is the overlap of irrelevant with relevant tuning}$

which reveals that the effects of noise, relevant tuning, and irrelevant tuning factorize (corresponding to the first, second, and third terms, respectively), and that – intuitively – μ_z and thus performance increases with γ and decreases with α, $\beta$, and ${\sigma }^2$ (where we always consider the latter a constraint and thus fixed, see the problem definition).

Note that in the small noise limit, ${\sigma }^2\ll {\alpha }^2$:

(S21) ${\mu }_z=\frac{1}{2{\sigma }^2}{\gamma }^2\left(1-{\beta }^2\right)$

showing that performance in this case can only be increased by decreasing $\beta$ (or, trivially, by increasing γ), but not by decreasing α. In contrast, when the small noise limit does not hold, the original Equation S17 applies, and so performance can be increased by decreasing either α or $\beta$ (or, again, by increasing γ).

Metabolic cost

Following previous work (Wójcik, 2023; Rigotti et al., 2013; Yang et al., 2019; Wang et al., 2018; Silver et al., 2016; Jensen et al., 2023), we define the metabolic cost to be the average sum of squared neural responses (where the averaging is over realizations of the relevant and irrelevant features as well as neural noise):

(S22) ${\omega }^2=\mathbb{E}\left[{\Vert \mathit{r}\Vert }^2\right]$

which, by making the averaging over relevant features explicit, can be rewritten as

(S23) ${\omega }^2=\frac{\mathbb{E}\left[{\Vert \mathbf{r}\Vert }^2|c_1\right]+\mathbb{E}\left[{\Vert \mathbf{r}\Vert }^2|c_2\right]}{2}$

where the metabolic cost for c₁ is

(S24) $\mathbb{E}\left[{\Vert \mathbf{r}\Vert }^2|c_1\right]=\mathbb{E}{\left[\mathbf{r}|c_1\right]}^{\mathrm{\top }}\mathbb{E}\left[\mathbf{r}|c_1\right]+\mathrm{T}\mathrm{r}\left(\mathbb{C}\left[\mathbf{r}|c_1\right]\right)$

(S25) $={\left(\mathit{\mu }+\frac{\mathrm{\Delta }{\mathbf{r}}_{\mathrm{r}\mathrm{e}\mathrm{l}}}{2}\right)}^{\mathrm{\top }}\left(\mathit{\mu }+\frac{\mathrm{\Delta }{\mathbf{r}}_{\mathrm{r}\mathrm{e}\mathrm{l}}}{2}\right)+\mathrm{T}\mathrm{r}\left(\mathbf{\Sigma }\right)$

(S26) $={\Vert \mathit{\mu }\Vert }^2+\frac{{\gamma }^2}{4}+{\mathit{\mu }}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}+\frac{{\alpha }^2}{4}+N{\sigma }^2$

and, again by symmetry,

(S27) $\mathbb{E}\left[{\Vert \mathbf{r}\Vert }^2|c_2\right]={\Vert \mathit{\mu }\Vert }^2+\frac{{\gamma }^2}{4}-{\mathit{\mu }}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}+\frac{{\alpha }^2}{4}+N{\sigma }^2$

and so, by substituting Equations S26 and S27 into Equation S23, we get:

(S28) ${\omega }^2={\Vert \mathit{\mu }\Vert }^2+N{\sigma }^2+\frac{{\gamma }^2}{4}+\frac{{\alpha }^2}{4}$

This result is also interesting, because it shows that the metabolic cost factorizes into four additive terms, each of which scales monotonically with a separate parameter: the average magnitude of responses, neural noise, the magnitude of relevant tuning, and the magnitude of irrelevant coding. It is also interesting to note what the metabolic cost does not depend on: the overlap between relevant and irrelevant tuning, $\beta$.

The first two terms in Equation S28 are assumed to be fixed (see problem definition, Problem definition), so we will not consider them further. The last two terms in Equation S28 increase with γ and α, respectively. As we want the metabolic cost to be small, this prefers small γ and α.

Note that, unlike for deriving the optimal linear decoder (The optimal linear decoder) and its performance (The performance of the optimal linear decoder), for which we needed to assume that $ϵ$ is normally distributed, no such assumption needed to be made for computing the metabolic cost (Equation S28). Thus, for the metabolic cost, the same result is obtained for example when the irrelevant feature is binary, such that $ϵ=\pm 1$ with equal probability (without loss of generality; note that, in this case, it is still true that ${\displaystyle \mathbb{E}\left[ϵ\right]=0}$ and ${\displaystyle \mathbb{V}\left[ϵ\right]=1}$, which is all that we assumed in the derivation above (see Equation S13)), i.e., when variability in the irrelevant feature simply changes the responses by ${\displaystyle \pm \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}$.

Qualitative predictions about optimal parameters

In general, the optimal setting of optimizable parameters, ${\displaystyle \mathrm{\Delta }{\mathbf{r}}_{\text{rel}}}$ and ${\displaystyle \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}$ (Problem definition), depends on the overall objective function, which will usually be a sum of performance (which we want to be high) and (negative) metabolic cost (which we want to be low), with some suitable Lagrange multiplier, λ, controlling the trade-off between the two terms:

(S29) $\mathcal{L}\left(\gamma ,\alpha ,\beta \right)=\underset{\text{Eqs. S11 and S17}}{\underbrace{\mathrm{\Pi }}}-\lambda \underset{\text{Eq. S28}}{\underbrace{{\omega }^2}}$

We note that the objective function also depends on the constraint parameters, µ and σ (Problem definition). The effect of µ is trivial: it only affects the metabolic cost, not performance, and it does so as a simple additive term (Equation S28), so it does not affect the optimal values of the other parameters. (This remains true even if µ is optimizable, in which case it is also obvious from Equation S28 that its optimal value is 0.) The effect of the other constraint, σ, is more nuanced, so we will separately consider two different regimes for it: small noise ($0\le \sigma \le {\sigma }_{\text{crit}}$) and large noise (${\displaystyle {\sigma }_{\text{crit}}<\sigma }$) with the constant ${\sigma }_{\text{crit}}$ defined later.

We also note that both performance (Equation S17) and metabolic cost (Equation S28) only depend on the optimizable parameters, ${\displaystyle \mathrm{\Delta }{\mathbf{r}}_{\text{rel}}}$ and ${\displaystyle \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}$, through their summary statistics, γ, α, and (for performance) $\beta$ (Equations S18–S20). In general, the dependence on the latter two parameters is straightforward: performance decreases in both α and $\beta$ (so that the last term in Equation S17 achieves its maximal possible value 1 when either parameter is 0), while the metabolic cost increases with α. This means that their optimal values will be at 0. However, we recall that in the small noise limit, only decreasing $\beta$ can improve performance (Equation S21), while otherwise, decreasing α can also make a contribution to it. At the same time, the metabolic cost only depends on α but not on $\beta$. Thus to summarize, in the small noise limit, there is ‘pressure’ on both $\beta$ (from performance) and α (from the metabolic cost) to be small. In other cases, the pressure on α comes from both performance and metabolic cost, while $\beta$ only matters for performance, so we expect α to be more aggressively minimized by optimization, and perhaps $\beta$ not so much (which we find to be the case for our optimized recurrent neural networks; Figure 4f and g; see also Curvature of the loss function around the optimum).

The overall effect of γ is slightly less trivial. Equation S28 shows that the metabolic cost depends on it quadratically, i.e., it should be small. However, as we saw earlier (Equation S17), decoding performance monotonically grows with it. Nevertheless, this dependence of decoding performance on γ is nonlinear (through the standard normal c.d.f., Equation S11), such that it is effectively linear in γ for small values, but saturates for large values of γ. (This is because ${\displaystyle \mathrm{\Psi }\left(x\right)}$ is linear in the x→0 limit, and the argument of ${\displaystyle \mathrm{\Psi }(\cdot )}$ for computing the performance is linear in the square root of μ_z (Equation S11), which in turn is quadratic in γ (Equation S17), so the argument of ${\displaystyle \mathrm{\Psi }(\cdot )}$ is linear in γ.) This means that for small values of γ the linear performance will dominate over the quadratic metabolic cost, but eventually, for large values, the quadratic metabolic term is guaranteed to dominate over the saturating performance, thus effectively limiting the magnitude of relevant tuning to some finite value. Note, however, that this argument does not yet reveal what happens to γ depending on the noise regime, and where the transition between these two regimes happens.

Fortunately, it is possible to analytically derive ${\gamma }_{*}$, the optimal value of γ, as a function of σ. For this, we assume that α and $\beta$ already take their optimal values (without loss of generality, as we are interested in jointly optimizing all relevant parameters), such that the last term in Equation S17 is simply 1 (see above). In this case, the terms in the overall objective function (Equation S29) that depend on the two remaining parameters of interest (γ and σ) are simply:

(S30) $\mathcal{L}=\mathrm{\Psi }\left(\frac{\gamma }{2\sigma }\right)-\frac{\lambda }{4}{\gamma }^2-\lambda N{\sigma }^2+\dots$

The optimal γ can be simply defined implicitly as a function of σ as the solution to the following equation:

(S31) $0=\frac{\mathrm{\partial }\mathcal{L}}{\mathrm{\partial }\gamma }\left({\gamma }_{\ast },\sigma \right)$

We now substitute (the partial derivative of) Equation S30 into Equation S31 to obtain:

(S32) $0=\frac{1}{2\sigma }\mathcal{N}\left(\frac{{\gamma }_{\ast }}{2\sigma }\right)-\frac{\lambda }{2}{\gamma }_{\ast }$

Note that this yields a solution for any ${\displaystyle \sigma ,\lambda >0}$ since the line $\frac{\lambda }{2}{\gamma }_{*}$ intersects the pdf of the Normal distribution for some ${\gamma }_{*}\ge 0$. Re-arranging Equation S32 gives us:

(S33) ${\gamma }_{\ast }{e}^{\frac{{\gamma }_{\ast }^2}{8{\sigma }^2}}=\frac{1}{\sqrt{2\pi }\sigma \lambda }$

This equation has a solution in terms of the Lambert W function (defined by its inverse as ${\displaystyle W^{-1}\left(x\right)=x{e}^x}$)

(S34) ${\gamma }_{\ast }=2\sigma \sqrt{W\left(\frac{1}{8\pi {\lambda }^2{\sigma }^4}\right)}$

where we only take the positive solution since γ can only be positive.

We can find the maximum of this function by taking the derivative of Equation S34 and setting it equal to 0. This gives:

(S35) $W\left(\frac{1}{8\pi {\lambda }^2{\sigma }_{\text{crit}}^4}\right)=1$

Using the fact that ${\displaystyle W\left(e\right)=1}$, we obtain:

(S36) ${\sigma }_{\text{crit}}=\frac{1}{{\left(8\pi {\lambda }^{2}e\right)}^{1/4}}$

Substituting this into Equation S34 gives:

(S37) ${\gamma }_{\text{crit}}=2{\sigma }_{\text{crit}}$

These results are shown in Appendix 1—figure 1, together with a numerical confirmation (see also Figure 4d).

Appendix 1—figure 1

Download asset Open asset

The qualitative dependence of ${\gamma }_{*}$ on σ is straightforward to interpret and intuitive. In the small noise regime, $0\le \sigma \le {\sigma }_{\text{crit}}$, γ needs to grow with σ so that the separation of the response distributions conditioned on c₁ and c₂ remains large enough to guarantee high performance. This growth starts at zero because for zero noise any infinitesimal separation between the mean responses for c = c₁ and c₂ makes for perfect performance. Thus, γ remains small, and so – as we saw above – the performance term in the objective function dominates over the metabolic cost term. This means that the value of γ that optimizes the full objective function can be understood from just this performance-based perspective. However, in the large noise regime, ${\displaystyle {\sigma }_{\text{crit}}<\sigma }$, γ would need to be so large to guarantee high performance that it would reach a regime in which – as we saw above – the metabolic cost dominates. Thus the optimal γ is increasingly influenced by the metabolic cost, and thus decreases with σ.

Finally, we derive classification performance with optimized parameters as a function of σ by substituting Equation S34 into Equations S11 and S17 and obtain:

(S38) $\mathrm{\Pi }=\mathrm{\Psi }\left(\sqrt{W\left(\frac{1}{8\pi {\lambda }^2{\sigma }^4}\right)}\right)$

which shows the intuitive result that performance monotonically decreases with σ and drops to chance level for $\sigma \to \infty$ (Appendix 1—figure 2, red). This is because the argument of W monotonically decreases with σ from infinity to zero, while both ${\displaystyle W\left(x\right)}$ and $\sqrt{x}$ monotonically increase (without bounds) with x from zero at x=0, and finally $\Psi \left(x\right)$ also monotonically increases with x but from $\frac{1}{2}$ at x=0 and to an asymptotic bound of 1.

Appendix 1—figure 2

Download asset Open asset

Curvature of the loss function around the optimum

The curvature of the loss landscape around the optimum is important for determining how tightly constrained the parameters will be in the presence of noisy gradient updates due to the finite batch sizes used for stochastic gradient descent. Before we proceed, we first repeat here the main equations regarding the loss ${\displaystyle \mathcal{L}}$:

(S39) $\mathcal{L}=\mathrm{\Pi }-\lambda {\omega }^2$

(S40) $\Pi =\Psi \left(\sqrt{\frac{{\mu }_z}{2}}\right)$

(S41) ${\mu }_z=\frac{1}{2{\sigma }^2}{\gamma }^2\left(1-\frac{{\alpha }^2}{4{\sigma }^2+{\alpha }^2}{\beta }^2\right)$

(S42) ${\omega }^2={\Vert \mathit{\mu }\Vert }^2+N{\sigma }^2+\frac{{\gamma }^2}{4}+\frac{{\alpha }^2}{4}$

Curvature of the loss function with respect to α

In this section, we are interested in the first non-zero term of the Taylor expansion of the loss function ${\displaystyle \mathcal{L}}$ with respect to α – the magnitude of the irrelevant input – around the optimum at $\alpha =\beta =0$. It turns out that the first non-zero term results from the second derivative of the metabolic cost term in the loss function since the other terms always contain at least an α or $\beta$ (which are 0 at the optimum). We therefore find that:

(S43) $\frac{{\mathrm{\partial }}^2\mathcal{L}}{\mathrm{\partial }{\alpha }^2}{|}_{\alpha =0,\beta =0}=\frac{{\mathrm{\partial }}^2\mathrm{\Pi }}{\mathrm{\partial }{\alpha }^2}{|}_{\alpha =0,\beta =0}-\lambda \frac{{\mathrm{\partial }}^2{\omega }^2}{\mathrm{\partial }{\alpha }^2}{|}_{\alpha =0,\beta =0}$

(S44) $=0-\lambda \frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\alpha }^2}\left[{\Vert \mathit{\mu }\Vert }^2+N{\sigma }^2+\frac{{\gamma }^2}{4}+\frac{{\alpha }^2}{4}\right]$

(S45) $=-\frac{1}{2}\lambda .$

From this result, we see that the magnitude of the curvature of the loss decreases as a function of λ. In other words, the loss landscape as a function of the irrelevant input α becomes steeper with increasing regularization. This is why we observed larger values of ${\displaystyle \alpha =\Vert \mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}\Vert }$ (Equation S18) in Figure 4f with decreasing λ.

Curvature of the loss function with respect to β

We now turn our attention to the first non-zero term of the Taylor expansion of the loss function ${\displaystyle \mathcal{L}}$ with respect to $\beta$ – the overlap of irrelevant with relevant tuning – around the optimum at $\alpha =\beta =0$. It turns out that the first non-zero term results from a fourth derivative of the performance term Π with respect to both α and $\beta$:

(S46) $\frac{{\mathrm{\partial }}^4\mathcal{L}}{\mathrm{\partial }{\alpha }^2{\beta }^2}{|}_{\alpha =0,\beta =0}=\frac{{\mathrm{\partial }}^4\mathrm{\Pi }}{\mathrm{\partial }{\alpha }^2{\beta }^2}{|}_{\alpha =0,\beta =0}-\lambda \frac{{\mathrm{\partial }}^4{\omega }^2}{\mathrm{\partial }{\alpha }^2{\beta }^2}{|}_{\alpha =0,\beta =0}$

(S47) $=\frac{{\mathrm{\partial }}^4\mathrm{\Pi }}{\mathrm{\partial }{\alpha }^2{\beta }^2}{|}_{\alpha =0,\beta =0}-\lambda \frac{{\mathrm{\partial }}^4}{\mathrm{\partial }{\alpha }^2{\beta }^2}\left[{\Vert \mathit{\mu }\Vert }^2+N{\sigma }^2+\frac{{\gamma }^2}{4}+\frac{{\alpha }^2}{4}\right]{|}_{\alpha =0,\beta =0}$

(S48) $=\frac{{\mathrm{\partial }}^4\mathrm{\Pi }}{\mathrm{\partial }{\alpha }^2{\beta }^2}{|}_{\alpha =0,\beta =0}-0.$

(S49) $=\frac{{\mathrm{\partial }}^4\mathrm{\Pi }}{\mathrm{\partial }{\alpha }^2{\beta }^2}{|}_{\alpha =0,\beta =0}$

To evaluate this term we note that:

(S50) $\frac{{\mathrm{\partial }}^2\mathrm{\Pi }}{\mathrm{\partial }{\beta }^2}=\frac{{\mathrm{\partial }}^2\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z^2}\frac{\mathrm{\partial }{\mu }_z}{\mathrm{\partial }\beta }+\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\frac{{\mathrm{\partial }}^2{\mu }_z}{\mathrm{\partial }{\beta }^2}$

(S51) $=\frac{{\mathrm{\partial }}^2\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z^2}\left(-\frac{{\gamma }^2}{2{\sigma }^2}\frac{2{\alpha }^2\beta }{4{\sigma }^2+{\alpha }^2}\right)+\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\left(-\frac{{\gamma }^2}{2{\sigma }^2}\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\right)$

Therefore,

(S52) $\frac{{\mathrm{\partial }}^4\mathrm{\Pi }}{\mathrm{\partial }{\alpha }^2{\beta }^2}{|}_{\alpha =0,\beta =0}=\frac{{\gamma }^2}{2{\sigma }^2}\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\alpha }^2}\left(\frac{{\mathrm{\partial }}^2\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z^2}\left(-\frac{2{\alpha }^2\beta }{4{\sigma }^2+{\alpha }^2}\right)+\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\left(-\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\right)\right){|}_{\alpha =0,\beta =0}$

(S53) $=\frac{\beta {\gamma }^2}{2{\sigma }^2}\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\alpha }^2}\left(\frac{{\mathrm{\partial }}^2\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z^2}\left(-\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\right)\right)+\frac{{\gamma }^2}{2{\sigma }^2}\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\alpha }^2}\left(\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\left(-\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\right)\right){|}_{\alpha =0,\beta =0}$

(S54) $=\frac{{\gamma }^2}{2{\sigma }^2}\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\alpha }^2}\left(\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\left(-\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\right)\right){|}_{\alpha =0,\beta =0}$

Now, noting that ${\displaystyle \frac{\mathrm{\partial }}{\mathrm{\partial }\alpha }=\frac{\mathrm{\partial }}{\mathrm{\partial }{\mu }_z}\frac{\mathrm{\partial }{\mu }_z}{\mathrm{\partial }\alpha }}$, we obtain:

(S55) $\begin{array}{l}\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\alpha }^2}\left(\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\left(-\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\right)\right){|}_{\alpha =0,\beta =0}=-\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\left(\frac{{\mathrm{\partial }}^2\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z^2}\frac{{\mathrm{\partial }}^2{\mu }_z}{\mathrm{\partial }{\alpha }^2}+\frac{{\mathrm{\partial }}^3\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z^3}{\left(\frac{{\mathrm{\partial }}^2{\mu }_z}{\mathrm{\partial }\alpha }\right)}^2\right)\\ +\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\alpha }^2}\left(-\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\right){|}_{\alpha =0,\beta =0}\end{array}$

(S56) $=\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\frac{{\mathrm{\partial }}^2}{\mathrm{\partial }{\alpha }^2}\left(-\frac{2{\alpha }^2}{4{\sigma }^2+{\alpha }^2}\right){|}_{\alpha =0,\beta =0}$

(S57) $=\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\frac{\mathrm{\partial }}{\mathrm{\partial }\alpha }\left(-\frac{4\alpha }{4{\sigma }^2+{\alpha }^2}+\frac{4{\alpha }^3}{{\left(4{\sigma }^2+{\alpha }^2\right)}^2}\right){|}_{\alpha =0,\beta =0}$

(S58) $=\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}\left(-\frac{4}{4{\sigma }^2+{\alpha }^2}+\frac{8\alpha }{{\left(4{\sigma }^2+{\alpha }^2\right)}^2}+\frac{12{\alpha }^2}{{\left(4{\sigma }^2+{\alpha }^2\right)}^2}-\frac{16{\alpha }^4}{{\left(4{\sigma }^2+{\alpha }^2\right)}^3}\right){|}_{\alpha =0,\beta =0}$

(S59) $=-\frac{4}{4{\sigma }^2}\frac{\mathrm{\partial }\mathrm{\Pi }}{\mathrm{\partial }{\mu }_z}{|}_{\alpha =0,\beta =0}$

(S60) $=-\frac{4}{4{\sigma }^2}\frac{1}{4}\sqrt{\frac{2}{{\mu }_z}}\mathcal{N}\left(\sqrt{\frac{{\mu }_z}{2}}\right){|}_{\alpha =0,\beta =0}$

(S61) $=-\frac{1}{4{\sigma }^2}\frac{2\sigma }{\gamma }\mathcal{N}\left(\frac{\gamma }{2\sigma }\right)$

(S62) $=-\frac{1}{2\sigma \gamma }\mathcal{N}\left(\frac{\gamma }{2\sigma }\right)$

Therefore, from Equation S54 we obtain:

(S63) $\frac{{\mathrm{\partial }}^4\mathcal{L}}{\mathrm{\partial }{\alpha }^2{\beta }^2}{|}_{\alpha =0,\beta =0}=-\frac{\gamma }{4{\sigma }^3}\mathcal{N}\left(\frac{\gamma }{2\sigma }\right)$

We therefore find that the magnitude of this term becomes smaller with increasing noise. This is why we observed larger values of ${\displaystyle \beta =\frac{\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}^{\mathrm{\top }}\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}}{||\mathrm{\Delta }{\mathbf{r}}_{\text{rel}}||||\mathrm{\Delta }{\mathbf{r}}_{\text{irrel}}||}}$ (Equation S20) in Figure 4g with increasing σ.

All code was custom written in Python using NumPy, SciPy, Matplotlib, Scikit-learn, and Tensorflow. All code is available in GitHub (copy archived at Stroud, 2023). All neural recordings have been deposited at Dryad.

1. 1. Wojic M
   2. Stroud J
   3. Wasmuht D
   4. Kusunoki M
   5. Kadohisa M
   6. Buckley M
   7. Myers N
   8. Hunt L
   9. Duncan J
   10. Stokes M

   (2024) Dryad Digital Repository

   Electrophysiological recordings of prefrontal activity over learning in non-human primates.

   https://doi.org/10.5061/dryad.c2fqz61kb

1. 1. Asaad WF
   2. Rainer G
   3. Miller EK

   (2000) Task-specific neural activity in the primate prefrontal cortex

   Journal of Neurophysiology 84:451–459.

   https://doi.org/10.1152/jn.2000.84.1.451

   • PubMed
   • Google Scholar
2. 1. Barak O
   2. Sussillo D
   3. Romo R
   4. Tsodyks M
   5. Abbott LF

   (2013) From fixed points to chaos: three models of delayed discrimination

   Progress in Neurobiology 103:214–222.

   https://doi.org/10.1016/j.pneurobio.2013.02.002

   • PubMed
   • Google Scholar
3. 1. Bartolo R
   2. Saunders RC
   3. Mitz AR
   4. Averbeck BB

   (2020) Dimensionality, information and learning in prefrontal cortex

   PLOS Computational Biology 16:e1007514.

   https://doi.org/10.1371/journal.pcbi.1007514

   • PubMed
   • Google Scholar
4. 1. Bernardi S
   2. Benna MK
   3. Rigotti M
   4. Munuera J
   5. Fusi S
   6. Salzman CD

   (2020) The geometry of abstraction in the hippocampus and prefrontal cortex

   Cell 183:954–967.

   https://doi.org/10.1016/j.cell.2020.09.031

   • PubMed
   • Google Scholar
5. 1. Braver TS

   (2012) The variable nature of cognitive control: A dual mechanisms framework

   Trends in Cognitive Sciences 16:106–113.

   https://doi.org/10.1016/j.tics.2011.12.010

   • Google Scholar
6. 1. Chadwick A
   2. Khan AG
   3. Poort J
   4. Blot A
   5. Hofer SB
   6. Mrsic-Flogel TD
   7. Sahani M

   (2023) Learning shapes cortical dynamics to enhance integration of relevant sensory input

   Neuron 111:106–120.

   https://doi.org/10.1016/j.neuron.2022.10.001

   • PubMed
   • Google Scholar
7. 1. Churchland MM
   2. Yu BM
   3. Ryu SI
   4. Santhanam G
   5. Shenoy KV

   (2006) Neural variability in premotor cortex provides a signature of motor preparation

   The Journal of Neuroscience 26:3697–3712.

   https://doi.org/10.1523/JNEUROSCI.3762-05.2006

   • PubMed
   • Google Scholar
8. 1. Costa RP
   2. Padamsey Z
   3. D’Amour JA
   4. Emptage NJ
   5. Froemke RC
   6. Vogels TP

   (2017) Synaptic transmission optimization predicts expression loci of long-term plasticity

   Neuron 96:177–189.

   https://doi.org/10.1016/j.neuron.2017.09.021

   • PubMed
   • Google Scholar
9. Conference

   1. Cueva CJ
   2. Wei XX

   (2018)

   Emergence of grid-like representations by training recurrent neural networks to perform spatial localization

   6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings. pp. 1–19.

   • Google Scholar
10. 1. Cueva CJ
    2. Saez A
    3. Marcos E
    4. Genovesio A
    5. Jazayeri M
    6. Romo R
    7. Salzman CD
    8. Shadlen MN
    9. Fusi S

    (2020) Low-dimensional dynamics for working memory and time encoding

    PNAS 117:23021–23032.

    https://doi.org/10.1073/pnas.1915984117

    • PubMed
    • Google Scholar
11. 1. Deneve S
    2. Latham PE
    3. Pouget A

    (2001) Efficient computation and cue integration with noisy population codes

    Nature Neuroscience 4:826–831.

    https://doi.org/10.1038/90541

    • PubMed
    • Google Scholar
12. 1. DiCarlo JJ
    2. Cox DD

    (2007) Untangling invariant object recognition

    Trends in Cognitive Sciences 11:333–341.

    https://doi.org/10.1016/j.tics.2007.06.010

    • Google Scholar
13. Preprint

    1. Driscoll L
    2. Shenoy KV
    3. Sussillo D

    (2022) Flexible Multitask Computation in Recurrent Networks Utilizes Shared Dynamical Motifs

    bioRxiv.

    https://doi.org/10.1101/2022.08.15.503870

    • Google Scholar
14. 1. Dubreuil A
    2. Valente A
    3. Beiran M
    4. Mastrogiuseppe F
    5. Ostojic S

    (2022) The role of population structure in computations through neural dynamics

    Nature Neuroscience 25:783–794.

    https://doi.org/10.1038/s41593-022-01088-4

    • PubMed
    • Google Scholar
15. 1. Duncan J

    (2001) An adaptive coding model of neural function in prefrontal cortex

    Nature Reviews. Neuroscience 2:820–829.

    https://doi.org/10.1038/35097575

    • PubMed
    • Google Scholar
16. 1. Durstewitz D
    2. Vittoz NM
    3. Floresco SB
    4. Seamans JK

    (2010) Abrupt transitions between prefrontal neural ensemble states accompany behavioral transitions during rule learning

    Neuron 66:438–448.

    https://doi.org/10.1016/j.neuron.2010.03.029

    • PubMed
    • Google Scholar
17. 1. Echeveste R
    2. Aitchison L
    3. Hennequin G
    4. Lengyel M

    (2020) Cortical-like dynamics in recurrent circuits optimized for sampling-based probabilistic inference

    Nature Neuroscience 23:1138–1149.

    https://doi.org/10.1038/s41593-020-0671-1

    • PubMed
    • Google Scholar
18. 1. Enel P
    2. Procyk E
    3. Quilodran R
    4. Dominey PF

    (2016) Reservoir computing properties of neural dynamics in prefrontal cortex

    PLOS Computational Biology 12:e1004967.

    https://doi.org/10.1371/journal.pcbi.1004967

    • PubMed
    • Google Scholar
19. 1. Epsztein J
    2. Brecht M
    3. Lee AK

    (2011) Intracellular determinants of hippocampal CA1 place and silent cell activity in a novel environment

    Neuron 70:109–120.

    https://doi.org/10.1016/j.neuron.2011.03.006

    • PubMed
    • Google Scholar
20. 1. Everling S
    2. Tinsley CJ
    3. Gaffan D
    4. Duncan J

    (2002) Filtering of neural signals by focused attention in the monkey prefrontal cortex

    Nature Neuroscience 5:671–676.

    https://doi.org/10.1038/nn874

    • PubMed
    • Google Scholar
21. 1. Faria-Pereira A
    2. Morais VA

    (2022) Synapses: the brain’s energy-demanding sites

    International Journal of Molecular Sciences 23:3627.

    https://doi.org/10.3390/ijms23073627

    • PubMed
    • Google Scholar
22. 1. Flesch T
    2. Juechems K
    3. Dumbalska T
    4. Saxe A
    5. Summerfield C

    (2022) Orthogonal representations for robust context-dependent task performance in brains and neural networks

    Neuron 110:1258–1270.

    https://doi.org/10.1016/j.neuron.2022.01.005

    • PubMed
    • Google Scholar
23. 1. Freedman DJ
    2. Riesenhuber M
    3. Poggio T
    4. Miller EK

    (2001) Categorical representation of visual stimuli in the primate prefrontal cortex

    Science 291:312–316.

    https://doi.org/10.1126/science.291.5502.312

    • PubMed
    • Google Scholar
24. 1. Fusi S
    2. Miller EK
    3. Rigotti M

    (2016) Why neurons mix: high dimensionality for higher cognition

    Current Opinion in Neurobiology 37:66–74.

    https://doi.org/10.1016/j.conb.2016.01.010

    • PubMed
    • Google Scholar
25. 1. Ganguli S
    2. Bisley JW
    3. Roitman JD
    4. Shadlen MN
    5. Goldberg ME
    6. Miller KD

    (2008) One-dimensional dynamics of attention and decision making in LIP

    Neuron 58:15–25.

    https://doi.org/10.1016/j.neuron.2008.01.038

    • PubMed
    • Google Scholar
26. 1. Harris JJ
    2. Jolivet R
    3. Attwell D

    (2012) Synaptic energy use and supply

    Neuron 75:762–777.

    https://doi.org/10.1016/j.neuron.2012.08.019

    • PubMed
    • Google Scholar
27. 1. Hasenstaub A
    2. Otte S
    3. Callaway E
    4. Sejnowski TJ

    (2010) Metabolic cost as a unifying principle governing neuronal biophysics

    PNAS 107:12329–12334.

    https://doi.org/10.1073/pnas.0914886107

    • PubMed
    • Google Scholar
28. 1. Heald JB
    2. Lengyel M
    3. Wolpert DM

    (2021) Contextual inference underlies the learning of sensorimotor repertoires

    Nature 600:489–493.

    https://doi.org/10.1038/s41586-021-04129-3

    • PubMed
    • Google Scholar
29. Preprint

    1. Jensen KT
    2. Hennequin G
    3. Mattar MG

    (2023) A Recurrent Network Model of Planning Explains Hippocampal Replay and Human Behavior

    bioRxiv.

    https://doi.org/10.1101/2023.01.16.523429

    • Google Scholar
30. Book

    1. Kandel ER

    (2000)

    Principles of Neural Science

    McGraw-hill.

    • Google Scholar
31. 1. Kao TC
    2. Sadabadi MS
    3. Hennequin G

    (2021) Optimal anticipatory control as A theory of motor preparation: A thalamo-cortical circuit model

    Neuron 109:1567–1581.

    https://doi.org/10.1016/j.neuron.2021.03.009

    • PubMed
    • Google Scholar
32. 1. King JR
    2. Dehaene S

    (2014) Characterizing the dynamics of mental representations: the temporal generalization method

    Trends in Cognitive Sciences 18:203–210.

    https://doi.org/10.1016/j.tics.2014.01.002

    • Google Scholar
33. Preprint

    1. Kingma DP
    2. Ba J

    (2014) Adam: A Method for Stochastic Optimization

    arXiv.

    https://arxiv.org/abs/1412.6980

    • Google Scholar
34. 1. Laughlin SB

    (2001) Energy as a constraint on the coding and processing of sensory information

    Current Opinion in Neurobiology 11:475–480.

    https://doi.org/10.1016/s0959-4388(00)00237-3

    • PubMed
    • Google Scholar
35. 1. Lindsay GW

    (2021) Convolutional neural networks as a model of the visual system: past, present, and future

    Journal of Cognitive Neuroscience 33:2017–2031.

    https://doi.org/10.1162/jocn_a_01544

    • PubMed
    • Google Scholar
36. Preprint

    1. Löwe AT

    (2023) Regularised Neural Networks Mimic Human Insight

    arXiv.

    https://arxiv.org/abs/2302.11351

    • Google Scholar
37. Preprint

    1. Luo X
    2. Mok RM
    3. Roads BD
    4. Love BC

    (2023) A Controller-Peripheral Architecture and Costly Energy Principle for Learning

    bioRxiv.

    https://doi.org/10.1101/2023.01.16.524194

    • Google Scholar
38. 1. Maass W
    2. Natschläger T
    3. Markram H

    (2002) Real-time computing without stable states: A new framework for neural computation based on perturbations

    Neural Computation 14:2531–2560.

    https://doi.org/10.1162/089976602760407955

    • PubMed
    • Google Scholar
39. 1. Mack ML
    2. Preston AR
    3. Love BC

    (2020) Ventromedial prefrontal cortex compression during concept learning

    Nature Communications 11:46.

    https://doi.org/10.1038/s41467-019-13930-8

    • PubMed
    • Google Scholar
40. 1. Mante V
    2. Sussillo D
    3. Shenoy KV
    4. Newsome WT

    (2013) Context-dependent computation by recurrent dynamics in prefrontal cortex

    Nature 503:78–84.

    https://doi.org/10.1038/nature12742

    • PubMed
    • Google Scholar
41. 1. Maris E
    2. Oostenveld R

    (2007) Nonparametric statistical testing of EEG- and MEG-data

    Journal of Neuroscience Methods 164:177–190.

    https://doi.org/10.1016/j.jneumeth.2007.03.024

    • Google Scholar
42. 1. Masse NY
    2. Yang GR
    3. Song HF
    4. Wang XJ
    5. Freedman DJ

    (2019) Circuit mechanisms for the maintenance and manipulation of information in working memory

    Nature Neuroscience 22:1159–1167.

    https://doi.org/10.1038/s41593-019-0414-3

    • PubMed
    • Google Scholar
43. 1. Mastrogiuseppe F
    2. Ostojic S

    (2018) Linking connectivity, dynamics, and computations in low-rank recurrent neural networks

    Neuron 99:609–623.

    https://doi.org/10.1016/j.neuron.2018.07.003

    • PubMed
    • Google Scholar
44. 1. Meyers EM
    2. Freedman DJ
    3. Kreiman G
    4. Miller EK
    5. Poggio T

    (2008) Dynamic population coding of category information in inferior temporal and prefrontal cortex

    Journal of Neurophysiology 100:1407–1419.

    https://doi.org/10.1152/jn.90248.2008

    • PubMed
    • Google Scholar
45. 1. Miller EK
    2. Cohen JD

    (2001) An integrative theory of prefrontal cortex function

    Annual Review of Neuroscience 24:167–202.

    https://doi.org/10.1146/annurev.neuro.24.1.167

    • PubMed
    • Google Scholar
46. 1. Monsell S

    (2003) Task switching

    Trends in Cognitive Sciences 7:134–140.

    https://doi.org/10.1016/S1364-6613(03)00028-7

    • Google Scholar
47. Conference

    1. Musslick S

    (2017)

    Multitasking Capability Versus Learning Efficiency in Neural Network Architectures

    CogSci 2017 - Proceedings of the 39th Annual Meeting of the Cognitive Science Society. pp. 829–834.

    • Google Scholar
48. 1. Orhan AE
    2. Ma WJ

    (2019) A diverse range of factors affect the nature of neural representations underlying short-term memory

    Nature Neuroscience 22:275–283.

    https://doi.org/10.1038/s41593-018-0314-y

    • PubMed
    • Google Scholar
49. 1. Parthasarathy A
    2. Herikstad R
    3. Bong JH
    4. Medina FS
    5. Libedinsky C
    6. Yen S-C

    (2017) Mixed selectivity morphs population codes in prefrontal cortex

    Nature Neuroscience 20:1770–1779.

    https://doi.org/10.1038/s41593-017-0003-2

    • PubMed
    • Google Scholar
50. 1. Rainer G
    2. Asaad WF
    3. Miller EK

    (1998) Selective representation of relevant information by neurons in the primate prefrontal cortex

    Nature 393:577–579.

    https://doi.org/10.1038/31235

    • PubMed
    • Google Scholar
51. 1. Rainer G
    2. Miller EK

    (2002) Timecourse of object-related neural activity in the primate prefrontal cortex during a short-term memory task

    The European Journal of Neuroscience 15:1244–1254.

    https://doi.org/10.1046/j.1460-9568.2002.01958.x

    • PubMed
    • Google Scholar
52. 1. Reinert S
    2. Hübener M
    3. Bonhoeffer T
    4. Goltstein PM

    (2021) Mouse prefrontal cortex represents learned rules for categorization

    Nature 593:411–417.

    https://doi.org/10.1038/s41586-021-03452-z

    • PubMed
    • Google Scholar
53. 1. Richards BA
    2. Lillicrap TP
    3. Beaudoin P
    4. Bengio Y
    5. Bogacz R
    6. Christensen A
    7. Clopath C
    8. Costa RP
    9. de Berker A
    10. Ganguli S
    11. Gillon CJ
    12. Hafner D
    13. Kepecs A
    14. Kriegeskorte N
    15. Latham P
    16. Lindsay GW
    17. Miller KD
    18. Naud R
    19. Pack CC
    20. Poirazi P
    21. Roelfsema P
    22. Sacramento J
    23. Saxe A
    24. Scellier B
    25. Schapiro AC
    26. Senn W
    27. Wayne G
    28. Yamins D
    29. Zenke F
    30. Zylberberg J
    31. Therien D
    32. Kording KP

    (2019) A deep learning framework for neuroscience

    Nature Neuroscience 22:1761–1770.

    https://doi.org/10.1038/s41593-019-0520-2

    • PubMed
    • Google Scholar
54. 1. Rigotti M
    2. Barak O
    3. Warden MR
    4. Wang X-J
    5. Daw ND
    6. Miller EK
    7. Fusi S

    (2013) The importance of mixed selectivity in complex cognitive tasks

    Nature 497:585–590.

    https://doi.org/10.1038/nature12160

    • PubMed
    • Google Scholar
55. Preprint

    1. Schaeffer R
    2. Khona M
    3. Fiete IR

    (2022) No free lunch from deep learning in neuroscience: a case study through models of the entorhinal-hippocampal circuit

    bioRxiv.

    https://doi.org/10.1101/2022.08.07.503109

    • Google Scholar
56. Preprint

    1. Schimel M
    2. Kao TC
    3. Hennequin G

    (2023) When and Why Does Motor Preparation Arise in Recurrent Neural Network Models of Motor Control?

    bioRxiv.

    https://doi.org/10.1101/2023.04.03.535429

    • Google Scholar
57. Conference

    1. Schuessler F
    2. Mastrogiuseppe F
    3. Dubreuil A
    4. Ostojic S
    5. Barak O

    (2020)

    The interplay between randomness and structure during learning in RNNs

    Advances in Neural Information Processing Systems. pp. 13352–13362.

    • Google Scholar
58. 1. Silver D
    2. Huang A
    3. Maddison CJ
    4. Guez A
    5. Sifre L
    6. van den Driessche G
    7. Schrittwieser J
    8. Antonoglou I
    9. Panneershelvam V
    10. Lanctot M
    11. Dieleman S
    12. Grewe D
    13. Nham J
    14. Kalchbrenner N
    15. Sutskever I
    16. Lillicrap T
    17. Leach M
    18. Kavukcuoglu K
    19. Graepel T
    20. Hassabis D

    (2016) Mastering the game of Go with deep neural networks and tree search

    Nature 529:484–489.

    https://doi.org/10.1038/nature16961

    • PubMed
    • Google Scholar
59. 1. Sohn H
    2. Narain D
    3. Meirhaeghe N
    4. Jazayeri M

    (2019) Bayesian computation through cortical latent dynamics

    Neuron 103:934–947.

    https://doi.org/10.1016/j.neuron.2019.06.012

    • PubMed
    • Google Scholar
60. 1. Song HF
    2. Yang GR
    3. Wang XJ

    (2016) Training excitatory-inhibitory recurrent neural networks for cognitive tasks: a simple and flexible framework

    PLOS Computational Biology 12:e1004792.

    https://doi.org/10.1371/journal.pcbi.1004792

    • PubMed
    • Google Scholar
61. 1. Stokes MG
    2. Kusunoki M
    3. Sigala N
    4. Nili H
    5. Gaffan D
    6. Duncan J

    (2013) Dynamic coding for cognitive control in prefrontal cortex

    Neuron 78:364–375.

    https://doi.org/10.1016/j.neuron.2013.01.039

    • PubMed
    • Google Scholar
62. Preprint

    1. Stroud JP
    2. Watanabe K
    3. Suzuki T
    4. Stokes MG
    5. Lengyel M

    (2021) Optimal information loading into working memory in prefrontal cortex explains dynamic coding

    bioRxiv.

    https://doi.org/10.1101/2021.11.16.468360

    • Google Scholar
63. Software

    1. Stroud JP

    (2023) Optimizing_RNNs_XOR_task, version swh:1:rev:f24ffb82d6bb813a0a059339dd538adea54c9c8d

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:72f5ef312797fc3afefb9f0bc5fc60f6e7697adc;origin=https://github.com/jakepstroud/optimizing_RNNs_XOR_task;visit=swh:1:snp:27503d476c06c68d1dcd4b092817a04a0bcc9399;anchor=swh:1:rev:f24ffb82d6bb813a0a059339dd538adea54c9c8d
64. 1. Sussillo D
    2. Churchland MM
    3. Kaufman MT
    4. Shenoy KV

    (2015) A neural network that finds A naturalistic solution for the production of muscle activity

    Nature Neuroscience 18:1025–1033.

    https://doi.org/10.1038/nn.4042

    • PubMed
    • Google Scholar
65. 1. Tomko GJ
    2. Crapper DR

    (1974) Neuronal variability: non-stationary responses to identical visual stimuli

    Brain Research 79:405–418.

    https://doi.org/10.1016/0006-8993(74)90438-7

    • PubMed
    • Google Scholar
66. 1. Wang J
    2. Narain D
    3. Hosseini EA
    4. Jazayeri M

    (2018) Flexible timing by temporal scaling of cortical responses

    Nature Neuroscience 21:102–110.

    https://doi.org/10.1038/s41593-017-0028-6

    • PubMed
    • Google Scholar
67. Preprint

    1. Whittington JCR
    2. Dorrell W
    3. Ganguli S
    4. Behrens TEJ

    (2022) Disentangling with Biological Constraints: A Theory of Functional Cell Types

    arXiv.

    https://arxiv.org/abs/2210.01768

    • Google Scholar
68. Preprint

    1. Wójcik MJ

    (2023) Learning Shapes Neural Dimensionality in the Prefrontal Cortex

    bioRxiv.

    https://doi.org/10.1101/2023.04.24.538054

    • Google Scholar
69. 1. Yang GR
    2. Joglekar MR
    3. Song HF
    4. Newsome WT
    5. Wang XJ

    (2019) Task representations in neural networks trained to perform many cognitive tasks

    Nature Neuroscience 22:297–306.

    https://doi.org/10.1038/s41593-018-0310-2

    • PubMed
    • Google Scholar
70. 1. Zhu F
    2. Wang R
    3. Pan X
    4. Zhu Z

    (2019) Energy expenditure computation of a single bursting neuron

    Cognitive Neurodynamics 13:75–87.

    https://doi.org/10.1007/s11571-018-9503-3

    • PubMed
    • Google Scholar