We consider data from subjects performing a delayed estimation task (Figure 1—source data 1). We briefly summarize the paradigm and the main findings; a more detailed description can be found in Pertzov et al. (2017) Subjects view a display with several ($K$) differently colored and oriented bars that are subsequently removed for the storage (delay) period. Following the storage period, subjects were cued by one of the colored bars in the display, now randomly oriented, and asked to rotate it to its remembered orientation. Bar orientations in the display were drawn randomly from the uniform distribution over all angles (thus the range of orientations lies in the circular interval ${\displaystyle [0,\pi ]}$) and the report of the subject was recorded as an analog value, to allow for more detailed and quantitative comparisons with theory (van den Berg et al., 2012). Importantly, both the number of items ($K$) and the storage duration ($T$) were varied.

When only a single item had to be remembered, the length of the storage interval had no statistically significant influence on the distribution of responses over the intervals considered (Figure 1B, with different delays marked by different shades and line styles; errors $<10$ degrees, effect of delay: $F(3,36)=1.3,p=0.3$; errors between $30-50$ degrees: $F(3,36)=0.2,p=0.9$). By contrast, response accuracy degraded significantly with delay duration when there were 6 items in the stimulus (Figure 1C; true orientation subtracted from all responses to provide a common center at 0 degrees). The number of very precise responses decreased (errors $<10$ degrees, effect of delay: $F(3,36)=6.15,p=0.002$), with a corresponding increase in the number of trials with large errors (e.g. errors between $30-50$ degrees, effect of delay: $F(3,36)=5.4,p=0.004$).

Figure 1 with 1 supplement see all

Download asset Open asset

Figure 1—source data 1

Experiment data used in the manuscript.

Subjects view a display with several ($K$) differently colored and oriented bars that are subsequently removed for the storage (delay) period. Following the storage period, subjects were cued by one of the colored bars in the display, now randomly oriented, and asked to rotate it to its remembered orientation. Bar orientations in the display were drawn randomly from the uniform distribution over all angles (thus the range of orientations lies in the circular interval $\mathrm{[}\mathrm{0}\mathrm{,}\pi \mathrm{]}$) and the report of the subject was recorded as an analog value. (See also [Pertzov et al., 2017]).

https://doi.org/10.7554/eLife.22225.004

Download elife-22225-fig1-data1-v2.zip

Overall, the squared error in recalling an item’s orientation (Figure 1D), averaged over subjects, increased with delay duration ($F(3,27)=49,p<0.001$) and also with item number ($F(3,27)=48,p<0.001$). The data show a clear interaction between storage interval duration and set size ($F(9,81)=17,p<0.001$), apparent as steeper degradation slopes for larger set-sizes. In summary, for a small number of items (e.g. $K=1,2$), increasing the storage duration does not strongly affect performance, but for any fixed delay, increasing item number has a more profound effect.

Finally, at all tested delays and item numbers, the squared errors are much smaller than the squared range of the circular variable, and any sub-linearities in the curves cannot be attributed to the inevitable saturation of a growing variance on a circular domain (Figure 1—figure supplement 1).

In this and all following sections, we start from the hypothesis that persistent neural activity underlies short-term information storage in the brain. The hypothesis is founded on evidence of a relationship between the stored variable and specific patterns of elevated (or depressed) neural activity (Taube, 1998; Aksay et al., 2001) that persist into the memory storage period and terminate when the task concludes, and on findings that fluctuations in delay-period neural activity can be predictive of variations in memory performance (Funahashi, 2006; Smith and Jonides, 1998; Blair and Sharp, 1995; Miller et al., 1996; Romo et al., 1999; Supèr et al., 2001; Harrison and Tong, 2009; Wimmer et al., 2014).

Neural network models like the ring attractor generate an activity bump that is a steady state of the network and thus persists when the input is removed, Figure 2A. All rotations of the canonical activity bump form a one-dimensional continuum of steady states, Figure 2B. Relatively straightforward extensions of the ring network can generate 2D or higher-dimensional manifolds of persistent states. However, any noise in network activity, for instance in form of stochastic spiking (Softky and Koch, 1993; Shadlen and Newsome, 1994), leads to lateral random drift along the manifold in the form of a diffusive (Ornstein-Uhlenbeck) random walk (Compte et al., 2000; Brody et al., 2003; Boucheny et al., 2005; Wu et al., 2008; Burak and Fiete, 2009; Fung et al., 2010; Burak and Fiete, 2012), Figure 2C–D.

Figure 2

Download asset Open asset

A defining feature of such random walks is that the squared deviation of the stored state relative to its initial value will grow linearly with elapsed time over short times, Figure 2D, with a proportionality constant $2𝒟$ (where $𝒟$ is the diffusivity) that depends on quantities like the size of the network and the peak firing rate of neurons (Burak and Fiete, 2012).

Suppose that the variables in a short-term memory task were directly transferred to persistent activity neural networks with a manifold of fixed points that matched the topology of the represented variable. Thus, $K$ circular variables would be stored, entry-by-entry, in $K$ 1-dimensional (1D) ring networks (Ben-Yishai et al., 1995). (Alternatively, the $K$ variables could be stored in a single network with a $K$-dimensional manifold of stable states, as described in the Appendix; the performance in neural costs and in fit to the data of this version of direct storage is worse than with storage in $K$ 1D networks, thus we focus on banks of 1D networks.)

When $N$ neural resources (e.g. composed of $N$ sets of $M$ neurons each, for a total of $NM$ neurons) are split into $K$ networks, each network is left with $N/K$ resources ($NM/K$ neurons in our example) for storage of a 1D variable. We know from (Burak and Fiete, 2012) that the diffusivity of the state in each of these 1D persistent activity networks will scale as the inverse of the number of neurons and of the peak firing rate per neuron. In other words, the diffusion coefficient is given by $\overline{𝒟}(K,N)=𝒟K/N$, where $𝒟$ is a diffusivity parameter independent of $K,N$ (but $𝒟\propto 1/M$). So long as the squared error remains small compared to the squared range of the variable, it will grow linearly in time at a rate given by $2\overline{𝒟}(K,N)$ (indeed, in the psychophysical data, the squared error remains small compared to the squared range of the angular variable; see Figure 1—figure supplement 1). Therefore the mean squared error (MSE) is given by:

The only free parameter in the expression for MSE as a function of time and item number is the ratio $N/2𝒟$. Because the inverse diffusivity parameter $1/𝒟$ scales with the number of neurons ($M$ in our example) when $N,K$ are held fixed, the product $N/(2𝒟)$ is proportional to the total number of neurons ($N/(2𝒟)\propto NM$). This ratio therefore functions as a combined neural resource parameter.

To fit the theory of direct storage to psychophysics data, we find a single best-fit value (with weighted least-squares) of the free parameter $N/2𝒟$ across all item numbers and storage durations. For each item number curve, the fits are additionally anchored to the shortest storage period point ($T=100$ ms), which serves as a proxy for baseline performance at zero delay. Such baseline errors close to zero delay – which may be due to limitations in sensory perception, attentional constraints, constraints on the rate of information encoding (loading) into memory, or other factors – are not the subject of the present study, which seeks to describe how performance will deteriorate over time relative to the zero-delay baseline, as a function of storage duration and item number.

As can be seen in Figure 3A, the direct storage theory provides a poor match to human memory performance ($p$ values that the data occur by sampling from the model, excluding the $100$ ms time-point: $0.07,0.38,<{10}^{-4}$ for 1 item; $0.39,<{10}^{-4},0.2$ for 2 items; $0.09,0.29,0.08$ for 4 items, and $<{10}^{-3},<{10}^{-4},<{10}^{-4}$ for 6). These $p$-values strongly suggest rejection of the model.

Figure 3 with 4 supplements see all

Download asset Open asset

Does the direct storage model fail mostly because its dependence on time and item number are linear, while the data exhibits some nonlinear effects at the largest delays? On the contrary, direct storage fails to fit the data even at short delays when the performance curves are essentially linear (see the systematic underestimation of squared error by the model over $\le 2$ second delays in the 4- and 6-item curves). If anything, the slight sub-linearity in the 6-item curve at longer delays tends to bring it closer to the other curves and thus to the model, thus its effect is to slightly reduce the discrepancy between the data and fits from direct storage theory.

One view of the results, obtained by selecting model parameters to best match the 6-item curve, is that direct storage theory predicts an insufficiently strong improvement in performance with decreasing item number, Figure 3B ($p$-values for direct-storage model when fit to the 6-item responses: $<{10}^{-3},{10}^{-3},<{10}^{-4}$ for 1 item; $<{10}^{-2},<{10}^{-4},<{10}^{-4}$ for 2 items; $0.76,<{10}^{-2},2\times {10}^{-3}$ for 4 items; $0.22,0.39,0.38$ for 6, excluding the $100$ ms delay time-point; the $p$-values for the 1- and 2-item curves strongly suggest rejection of the model).

Even if information storage in persistent activity networks is a central component of short-term memory, describing the storage step is not a sufficient account of memory. This fact is widely appreciated in memory psychophysics, where it has been observed that variations in attention, motivation, and other factors also affect memory performance (Atkinson and Shiffrin, 1968; Matsukura et al., 2007). Here we propose that, even discounting these complex factors, direct storage of a set of continuous variables into persistent activity networks with the same total dimension of stable states lacks generality as a model of memory because it does not consider how pre-encoding of information could affect its subsequent degradation, Figure 3C–E. This omission could help account for the mismatch between predictions from direct storage and human behavior, Figure 3A–B.

Storing information in noisy persistent activity networks means that after a delay there will be some information loss, as described above. Mathematically, information storage in a noisy medium is equivalent to passing the information through a noisy information channel. To allow for high-fidelity communication through a noisy channel, it is necessary to first appropriately encode the signal, Figure 3F. Encoding for error control involves the addition of appropriate forms of redundancy tailored to the channel noise. As shown by Shannon (Shannon, 1948), very different levels of accuracy can be achieved with different forms of encoding for the same amount of coding redundancy and channel noise. Thus, predictions for memory performance after good encoding may differ substantially from the predictions from direct storage even though the underlying storage networks (channels) are identical.

Thus, a more general theory of information storage for short-term memory in the brain would consider the effects of arbitrary encoder-decoder pairs that sandwich the noisy storage stage, Figure 3G. In such a three-stage model, information to be stored is first passed to an encoder, which performs all necessary encoding. Encoding strategies may include source coding or compression of the data as well as, critically, channel coding — the addition of redundancy tailored to the noise in the channel so that, subject to constraints on how much redundancy can be added, the downstream effects of channel noise are minimized (Shannon, 1948). The coded information is stored in persistent activity networks, Figure 3H. Finally, the information is accessed by a decoder or readout, Figure 3G. Here, we derive a bound on the best performance that can be achieved by any coding or decoding strategy, if the storage step involves graded persistent activity.

The encoder transforms the $K$-dimensional input variable into an $N$ dimensional codeword, to be stored in a bank of storage networks with an $N$-dimensional manifold of persistent activity states (in the form of $N$ networks with a 1-dimensional manifold each, or 1 network with an $N$-dimensional manifold, or something in between). To equalize resource use for the persistent activity networks in both direct storage and coded storage models of memory, the $N$ stored states have a diffusivity $𝒟$ each, in contrast to the diffusivity of $𝒟K/N$ each for $K$ states (compare Figure 3D–E and and G–H). The storage step is equivalent to passage of information through additive Gaussian information channels, with variance proportional to the storage duration $T$ and to the diffusivity. The decoder error-corrects the output of the storage stage and inverts the code to provide an estimate of the stored variable. (For more details, see Materials and methods and Appendix.)

We can use information theory to derive the minimum achievable recall error over all possible encoder-decoder structures, for the given statistics of the variable to be remembered and the noise in the storage information channels. In particular, we use joint source-channel coding theory to first consider at what rate information can be conveyed through a noisy channel for a given level of noise and coding redundancy, then obtain the minimal achievable distortion (recall error) for that information rate (see Materials and Appendix). We obtain the following lower-bound on the recall error:

This result is the theoretical lower bound on MSE achievable by any system that passes information through a noisy channel with the specified statistics: a Gaussian additive channel noise of zero mean and variance $2𝒟T$ per channel use, a codeword of dimension $N$, and a variable to be transmitted (stored) of dimension $K$, with entries that lie in the range $[0,\mathrm{\Phi }]$. The bound becomes tight asymptotically (for large $N$), but for small $N$ it remains a strict lower-bound. Although the potential for decoding errors is reduced at smaller $N$, the qualitative dependence of performance on item number and delay should remain the same (Appendix and (Polyanskiy et al., 2010) ). The bound is derived by dividing the total resources (defined here, as in the direct storage case, as the ratio $N/2𝒟$) evenly across all stored items (details in Appendix), similar to a ‘continuous resource’ conception of memory. The same theoretical treatment will admit different resource allocations, for instance, one could split the resources into a fixed number of pieces and allocate those to a (sub)set of the presented items, more similar to the ‘discrete slots’ model.

A heuristic derivation of the result above can be obtained by first noting that the capacity of a Gaussian channel with a given signal-to-noise ratio ($SNR$) is $I_{Gauss}=\frac{1}{2}\log (1+SNR)$. The summed capacity of $N$ channels, spread across the $K$ items of the stored variable, produces $I_{peritem}=\frac{N}{K}{I}_{Gauss}$. The variance of a scalar within the unit interval represented by $I$ bits of information is bounded below by $e^{-2I}$. Inserting $I_{peritem}$ into the variance expression and $SNR=1/2𝒟T$ into $I_{Gauss}$, yields Equation 2 , up to scaling prefactors. The Appendix provides more rigorous arguments that the bound we derive is indeed the best that can theoretically be achieved.

Equation 2 exhibits some characteristic features, including, first, a joint dependence on the number of stored items and the storage duration. According to this expression, the time-course of memory decay depends on the number of items. This effect arises because items compete for the same limited memory resources and when an item is allocated fewer resources it is more susceptible to the effects of noise over time. Second, the scaling with item number is qualitatively different than the scaling with storage duration: Increasing the number of stored items degrades performance much more steeply than increasing the storage interval, because item number is in the exponent. For a single memorized feature or item, the decline in accuracy with storage interval duration is predicted to be weak. On the other hand, increasing the number of memorized items while keeping the storage duration fixed should lead to a rapid deterioration in memory accuracy.

We next consider whether the performance of an optimal encoder (given this lower bound) can be distinguished from the direct storage model based on human performance data. The two predictions differ in their dependence upon the number of independent storage channels or networks, $N$, which we do not know how to control in human behavior. Equally important, since Equation 2 provides a theoretical limit on performance, it is of interest to learn whether human behavior approximates the limit, and where it might deviate from it.

In comparing the psychophysical data to the theoretical bound on short-term memory performance, there are two unknown parameters, $1/2𝒟$ (the inverse diffusivity in each persistent activity network) and $N$ (the number of such networks), both of which scale linearly with the neural resource of neuron number. The product of these parameters corresponds to total neural resource exactly as in the direct storage case. We fit Equation 2 to human performance data, assuming as in the direct storage model that the total neural resource is fixed across all item numbers and delay durations, and setting the 100 ms delay values of the theoretical curves to their empirical values.

The resulting best fit between theory and human behavior is excellent (Figure 4E; $p$ values that the data means may occur by sampling from the model, excluding the $T=100$ ms time-points: $0.99,0.07,0.75$ for 1 item; $0.46,0.07,0.60$ for 2 items; $0.54,0.24,0.43$ for 4 items; $0.89,0.38,0.32$ for 6; all values are larger than 0.05, most much more so. These $p$ values indicate a significantly better fit to data than obtained with the direct storage model).

Figure 4 with 2 supplements see all

Download asset Open asset

If we penalize the well-coded storage model for its extra parameter compared to direct storage ($1/2𝒟$ and $N$, versus the single parameter $𝒟/N$ for the direct storage model) through the Bayesian Information Criterion (BIC), a likelihood-based hypothesis comparison test (that more stringently penalizes model parameters than the AIC or Aikike Information Criterion), the evidence remains very strongly in favor of the well-coded memory storage model compared to direct storage (${\displaystyle \mathrm{\Delta }\mathrm{B}\mathrm{I}\mathrm{C}\approx 99\gg 10}$, where 10 is the cutoff for ‘very strong’ support) (Kass and Raftery, 1995). In fact, according to the BIC, the discrepancy in the quality of fit to the data between the models is so great that the increased parameter cost of the well-coded memory model barely perturbs the evidence in its favor. Some more statistical controls by jackknife cross-validation of the two models (Figure 3—figure supplement 1, Figure 3—figure supplement 2), exclusion of the $T=100$ ms point on the grounds that it might represent iconic memory recall rather than short-term memory (Figure 3—figure supplement 3), and redefinition of the number of items in memory to take into account the colors and orientations of the objects are given in the Appendix (Figure 3—figure supplement 4); the results are qualitatively unchanged, and also do not result in large quantitative deviations in the extracted parameters (discussed below).

The two-dimensional parameter space for fitting the theory to the data contains a one-dimensional manifold of reasonable solutions, Figure 4A (dark blue valley), most of which provide better fits to the data than the direct storage model. Some of these different fits to the data are shown in Figure 4B. At large values of $N$, the manifold is roughly a hyperbola in $\log N$ and $\log (1/2𝒟)$, suggesting that the logarithms of the two neural resource parameters can roughly trade off with each other; indeed, the total resource use in the one-dimensional solution valley is roughly constant at large $N$, Figure 4C (gray curves). However, at smaller $N$, the resource use drops with increasing $N$. The fits are not equally good along the valley of reasonable solutions, and the best fit lies near $N=5$ independent networks or channels (for jackknife cross-validation fits, see Figure 3—figure supplement 1, Figure 3—figure supplement 2, the best fits for the coded model can be closer to $N=10$; thus, the figure obtained for the number of memory networks should be taken as an order-of-magnitude estimate rather than an exact value). Resource use in the valley declines with increasing $N$ to its asymptotic constant value (thus larger $N$ would yield bigger representational efficiencies); however, by $N=5$, resource use is already close to its final asymptotic value, thus the gains of increasing the number of separate memory networks beyond ${\displaystyle N=5-10}$ diminish. The theory also provides good fits to individual subject performance for all ten subjects, using parameter values within a factor of 10 (and usually much less than a factor of 10) of each other (see Appendix).

Finally, we compare the neural resources required for storage in the direct storage model (best-fit) compared to the well-coded storage model. We quantify the neural resources required for well-coded storage as the product of the number of networks $N$ with the inverse diffusive coefficient $1/2𝒟$. This is proportional to the number of neurons required to implement storage. To replicate human behavior, coded storage requires resources totaling $N/2𝒟\approx 32$ (in units of seconds) for $N=5$, and $N/2𝒟\approx 22$ (s) for $N=10$, corresponding to the parameter settings for the fits in Figures 4C and 5B (center), respectively. By contrast, uncoded storage requires a 40-fold increase in $N$ or a 40-fold decrease in the diffusive growth rate in squared error, $2𝒟$, per network (or a corresponding increase in the product, $N/2𝒟$), because $N/2𝒟\approx 1215$ (s) under direct storage, to produce the best-fit result of Figure 3A. Thus, well-coded storage requires substantially fewer resources in the persistent activity networks for similar performance (assuming best fits of each produce similar performance). Equivalently, a memory system with good encoding can achieve substantially better performance with the same total storage resources, than if information were directly stored in persistent activity networks.

This result on the disparity in resource use between uncoded and coded information storage is an illustration of the power of strong error-correcting codes. Confronted with the prospect of imperfect information channels, finitely many resources, and the need to store or transmit information faithfully, one may take two different paths.

The first option is to split the total resources into $K$ storage bins, into which the $K$ variables are stored; when there are more variables, there are more bins and each variable receives a smaller bin. The other is to store $N$ quantities in $N$ bins regardless of $K$, by splitting each of the $K$ variables into $N$ pieces and assigning a piece from each of the different variables to one bin; when there are more variables, each variable gets a smaller piece of the bin. In the former approach, which is similar to the direct storage scenario, increasing $N$ would lead to improvements in the fidelity of each of the $K$ channels, Figure 4D. In the latter approach, which is the strong coding strategy, increasing $N$ would increase the number of channels while keeping their fidelity fixed, Figure 4B. The latter ultimately yields a more efficient use of the same total resources in terms of the final quality of performance, especially for larger values of $N$, at least without considering the cost of the encoding and decoding steps.

If we hold the total resource $N/2𝒟\propto NM$ fixed, the lowest achievable MSE (Equation 2 ) in the well-coded memory model is reached for maximally large $N$ and thus maximally large $𝒟$. However, human memory performance appears to be best-fit by $N\sim 10$. It is not clear, if our model does capture the basic architecture of the human memory system, why the memory system might operate in a regime of relatively small $N$. First, note that for increasing $N$, the total resource cost by $N=10$ is already down to within 10$\%$ of the minimum resource cost reached at much larger $N$. Second, note that the theory is derived under the ‘diffusive’ memory storage assumption: that within a storage network, information loss is diffusive. Thus, the assumption implicitly made while varying the parameter $N$ in Figure 4C is that as the number of networks ($N$) is increased, the diffusivity $𝒟\propto 1/M$ per network will simply increase in proportion to keep $NM$ fixed. However, the dynamics of persistent activity networks do not remain purely diffusive once the resource per network drops below a certain level: a new kind of non-diffusive error can start to become important (Schwab DJ & Fiete I (in preparation)). In this regime, the effective diffusivity in the network can grow much faster than the inverse network size. The non-diffusive errors produce large, non-local errors (which may be consistent with ‘pure guessing’ or ‘sudden death’ errors sometimes reported in memory psychophysics [Zhang and Luck, 2009]). It is possible that the memory networks operate in a regime where each channel (memory network) is allocated enough resources to mostly avoid non-diffusive errors, and this limits the number of networks.

We have provided a fundamental lower-bound on the error of recall in short-term memory as a function of item number and storage duration, if information is stored in graded persistent activity networks (our noisy channels). This bound on performance with an underlying graded persistent activity mechanism provides a reference point for comparison with human performance regardless of whether the brain employs strong encoding and decoding processes in its memory systems. The comparison can yield insights into the strategies the brain does employ.

Next, we used empirical data from analog measurements of memory error as a function of both temporal delay and the number of stored items. Using results from the theory of diffusion on continuous attractor manifolds in neural networks, we derived an expression for memory performance if the memorized variables were stored directly in graded persistent activity networks. The resulting predictions did not match human performance. The mismatch invites further investigation into whether and how direct-storage models can be modified to account for real memory performance.

Finally, we found that the bound from theory provided an (unexpectedly) good match to human performance, Figure 4. We are not privy to the actual values of the parameters $N,1/2𝒟$ in the brain and it is possible the brain uses a value of, to take an arbitrary example, $\approx 5\times N$ to achieve a performance reached with $N$ in Equation 2 , which would be (quantitatively) ‘suboptimal’. Nevertheless, the possibility that the brain might perform qualitatively according to the functional form of the theoretical bound is highly nontrivial: As we have seen, the addition of appropriate encoding and decoding systems can reduce the degradation in accuracy from scaling polynomially ($\sim 1/N$) in the number of neurons, as in direct storage, to scaling exponentially ($\sim e^{-\alpha N}$ for some $\alpha >0$). This is a startling possibility that requires more rigorous examination in future work.

Typical population codes for analog variables, as presently understood, exhibit linear gains in performance with $N$; such codes involve neurons with single-bump or ramp-like tuning curves that are offset or scaled copies of one another. For related reasons, persistent activity networks with such tuning curves also exhibit linear gains in memory performance with $N$ (Burak and Fiete, 2012). These ‘classical population codes’ are ubiquitous in the sensory and motor peripheries as well as some cognitive areas. So far, the only example of an analog neural code known in principle to be capable of exponential scaling with $N$ is the periodic, multi-scale code for location in grid cells of the mammalian entorhinal cortex (Hafting et al., 2005; Sreenivasan and Fiete, 2011; Mathis et al., 2012) : with this code, animals can represent an exponentially large set of distinct locations at a fixed local spatial resolution using linearly many neurons (Fiete et al., 2008; Sreenivasan and Fiete, 2011).

A literal analogy with grid cells would imply that all such codes should look periodic as a function of the represented variable, with a range of periods. A more general view is that the exponential capacity of the grid cell code results from two related features: First, no one group of grid cells with a common spatial tuning period carries full information about the coded variable (the spatial location of the animal) – location cannot be uniquely specified by the spatially periodic group response even in the absence of any noise. Second, the partial location information in different groups is independent because of the distinct spatial periods across groups (Sreenivasan and Fiete, 2011). In this more general view, strong codes need not be periodic, but there should be multiple populations that encode different, independent ‘parts’ of the same variable, which would be manifest as different sub-populations with diverse tuning profiles, and mixed selectivity to multiple variables.

It remains to be seen whether neural representations for short-term visual memory are consistent with strong codes. Intriguingly, neural responses for short-term memory are diverse and do not exhibit tuning that is as simple or uniform as typical for classical population codes (Miller et al., 1996; Fuster and Alexander, 1971; Romo et al., 1999; Wang, 2001; Funahashi, 2006; Fuster and Jervey, 1981; Rigotti et al., 2013). An interesting prediction of the well-coded model, amenable to experimental testing, is that the representation within a memory channel must be in an optimized format, and that this format is not necessarily the same format that information was initially presented in. The brain would have to perform a transformation from stimulus-space into a well-coded form, and one might expect to observe this transition of the representation at encoding. (See, e.g., recent works (Murray et al., 2017; Spaak et al., 2017), which show the existence of complex and heterogeneous dynamic transformations in primate prefrontal cortex during working memory tasks.) The less orthogonal the original stimulus space is to noise during storage and the more optimized the code for storage to resist degradation, the more different the mnemonic code will be from the sample-evoked signal. Studies that attempt to decode a stimulus from delay-period neural or BOLD activity on the basis of tuning curves obtained from the stimulus-evoked period are well-suited to test this question (Zarahn et al., 1999; Courtney et al., 1997; Pessoa et al., 2002; Jha and McCarthy, 2000; Miller et al., 1996; Baeg et al., 2003; Meyers et al., 2008; Stokes et al., 2013) : If it is possible to use early stimulus-evoked responses to accurately decode the stimulus over the delay-period (Zarahn et al., 1999; Courtney et al., 1997; Pessoa et al., 2002; Jha and McCarthy, 2000; Miller et al., 1996), it would suggest that information is not re-coded for noise resistance. On the other hand, a representation that is reshaped during the delay period relative to the stimulus-evoked response (Baeg et al., 2003; Meyers et al., 2008; Stokes et al., 2013) might support the possibility of re-coding for storage.

On the other hand, the encoding and decoding steps for strong codes add considerable complexity to the storage task, and it is unclear whether these steps can be performed efficiently so that the efficiencies of these codes are not nullified by their costs. In light of our current results, it will be interesting to further probe with neurophysiological tools whether storage for short-term visual memory is consistent with strong neural codes. With psychophysics, it will be important to compare human performance and the information-theoretic bound in greater detail. On the theoretical side, studying the decoding complexity of exponential neural codes is a topic of ongoing work (Fiete et al., 2014; Chaudhuri and Fiete, 2015), where we find that non-sparse codes made up of a product of many constraints on small subsets of the codewords might be amenable to strong error correction through simple neural dynamics.

Compared to other information-theoretic considerations of memory (Brady et al., 2009; Sims et al., 2012), the distinguishing feature of our approach is our focus on neuron- or circuit-level noise and the fundamental limits such noise will impose on persistence.

Our theoretical framework permits the incorporation of many additional elements: Variable allocation of resources during stimulus presentation based on task complexity, perceived importance, attention, and information loading rate, may all be incorporated into the present framework. This can be achieved by modeling $1/2𝒟$ and $N$ as dependent functions (e.g. as done in [van den Berg et al., 2012; Sims et al., 2012; Elmore et al., 2011]) rather than independent parameters, and by exploiting the flexibility allowed by our model in uneven resource allocation across items in the display (Materials and methods).

The memory psychophysics literature contains evidence of more complex memory effects, including a type of response called ‘sudden death’ or pure guessing (Zhang and Luck, 2009; Anderson et al., 2011). These responses are characterized by not being localized around the true value of the cued variable, and contribute a uniform or pedestal component to the response distribution. Other studies show that these apparent pedestals may not be a separate phenomenon and can, at least in some cases, be modeled by a simple growth in the variance over a bounded (circular) variable of a unimodal response distribution that remains centered at the cue location (van den Berg et al., 2012; Bays, 2014; Ma et al., 2014). In our framework, good encoding ensures that for noise below a threshold, the decoder can recover an improved estimate of the stored variable; however, strong codes exhibit sharp threshold behavior as the noise in the channel is varied smoothly. Once the noise per channel grows beyond the threshold, so-called catastrophic or threshold errors will occur, and the errors will become non-local: this phenomenon will look like sudden death in the memory report. In this sense, an optimal coding and decoding framework operating on top of continuously diffusing states in memory networks is consistent with the existence of sudden death or pure guessing-like responses, even without a distinct underlying mechanistic process in the memory networks themselves. We note, however, that the fits to the data shown here were all in the below-threshold regime.

Another complex effect in memory psychophysics is misbinding, in which one or more of the multiple features (color, orientation, size, etc.) of an item are mistakenly associated with those from another item. This work should be viewed as a model of single-feature memory. Very recently, there have been attempts to model misbinding (Matthey et al., 2015). It may be possible to extend the present model in the direction of (Matthey et al., 2015) by imagining the memory networks to be multi-dimensional attractors encoding multiple features of an item.

It will be important to understand whether in the direct coding model, modifications with plausible biological interpretations can lead to significantly better agreement with the data. From a purely curve-fitting perspective, the model requires stronger-than-linear improvement in recall accuracy with declining item number, and one might thus convert the combined resource parameter $N/𝒟$ in Equation 1 into a function that varies inversely with $K$. This step would result in a better fit, but would correspond in the direct storage model to an increased allocation of total memory resources when the task involves fewer items, an implausible modification. Alternatively, if multiple items are stored within a single persistent activity network, collision effects can limit performance for larger item numbers (Wei et al., 2012), but a quantitative result on performance as a function of delay time and item number remain to be worked out. Further examination of the types of data we have considered here, with respect to predictions that would result from a memory model dependent on direct storage of variables into persistent activity network(s), should help further the goal of linking short-term memory performance with neural network models of persistent activity.

Finally, note that our results stem from considering a specific hypothesis about the neural substrates of short-term memory (that memory is stored in a continuum of persistent activity states) and from the assumption that forgetting in short-term memory is undesirable but neural resources required to maintain information have a cost. It will also be interesting to consider the possibility of information storage in discrete rather than graded persistent activity states, with appropriate discretization of analog information before storage. Such storage networks will yield different bounds on memory performance than derived here (Koulakov et al., 2002; Goldman et al., 2003; Fiete et al., 2014), which should include the existence of small analog errors arising from discretization at the encoding stage, with little degradation over time because of the resistance of discrete states to noise. Also of great interest is to obtain predictions about degradation of short-term memory in activity-silent mechanisms such as synaptic facilitation (Barak and Tsodyks, 2014; Mi et al., 2017; Stokes, 2015; Lundqvist et al., 2016). A distinct alternate perspective on the limited persistence of short-term memory is that forgetting is a design feature that continually clears the memory buffer for future use and that limited memory allows for optimal search and computation that favors generalization instead of overfitting (Cowan, 2001). In this view, neural noise and resource constraints are not bottlenecks and there may be little imperative to optimize neural codes for greater persistence and capacity. To this end, it will be interesting to consider predictions from a theory in which limited memory is a feature, against the predictions we have presented here from the perspective that the neural system must work to avoid forgetting.

Ten neurologically normal subjects (age range $19$-$35$ yr) participated in the experiment after giving informed consent. All subjects reported normal or corrected-to-normal visual acuity. Stimuli were presented at a viewing distance of $60$ cm on a $21$” CRT monitor. Each trial began with the presentation of a central fixation cross (white, ${0.8}^{\circ }$ diameter) for $500$ milliseconds, followed by a memory array consisted of $1$, $2$, $4$, or $6$ oriented bars ($2^{\circ }\times {0.3}^{\circ }$ of visual angle) presented on a grey background on an imaginary circle (radius ${4.4}^{\circ }$) around fixation with equal inter-item distances (centre to centre). The colors of the bars in each trial were randomly selected out of eight easily-distinguishable colors. The stimulus display was followed by a blank delay of $0.1,1,2$ or $3$ seconds and at the end of each sequence, recall for one of the items was tested by displaying a ‘probe’ bar of the same color with a random orientation. Subjects were instructed to rotate the probe using a response dial (Logitech Intl. SA) to match the remembered orientation of the item of the same color in the sequence - henceforth termed the target. Each of the participants performed between $11$ and $15$ blocks of $80$ trials. Each block consisted of $20$ trials for each of the $4$ possible item numbers, consisting of $5$ trials for each delay duration.

In all fits of theory to data (for direct and well-coded storage), we assume that recall error at the shortest storage interval of 100 ms reflects baseline errors unrelated to the temporal loss of recall accuracy from noisy storage that is the focus of the present work. Under the assumption that this early (‘initial’) error is independent of the additional errors accrued over the storage period, it is appropriate to treat the baseline ($T=100$ ms) MSE as an additive contribution to the rest of the MSE (the variance of the sum of independent random variables is the sum of their variances). For this reason, we are justified in treating the $T=100$ ms errors as given by the data and setting these points as the initial offsets of the theory curves, which go on to explain the temporal (item-dependent) degradation of information placed in noisy storage.

The curves are fit by minimizing the summed weighted squared error of the theoretical prediction in fitting the subject-averaged performance data over all item numbers and storage durations. The theoretical predictions are given by Equation 1 for direct storage and Equation 2 for well-coded storage. The weights in the weighted least-squares are the inverse SEMs for each (item, storage duration) pair. The parameters of the fit are $N/2𝒟$ (direct storage model) or $N$ and $2𝒟$ (well-coded model). The parameter value selected is common across all item numbers and storage durations. The $p$ values given in the main paper quantify how likely the data means are to have been based on samples from a Gaussian distribution centered on the theoretical prediction.

The Bayesian Information Criterion (BIC) is a likelihood-based method for model comparison, with a penalty term that takes into account the number of parameters used in the candidate models. BIC is a Bayesian model comparison method, as discussed in Kass and Raftery (1995)

Given data $x$ that are (assumed to be) drawn from a distribution in the exponential family and a model $M(\overrightarrow{\theta })$ with associated parameters $\overrightarrow{\theta }$ ($\overrightarrow{\theta }$ is a vector of $k$ parameters), the BIC is given by:

where $n$ is the number of observations, and $\widehat{L}$ is the likelihood of the model (with parameters $\overrightarrow{\theta }$ selected by maximum likelihood). The smaller the BIC, the better the model. The more positive the difference

between a pair of models $M_1({\overrightarrow{\theta }}_1)$ and $M_2({\overrightarrow{\theta }}_2)$ (with associated parameters ${\overrightarrow{\theta }}_1,{\overrightarrow{\theta }}_2$, respectively, possibly of different dimensions $k_1\ne k_2$), the stronger the evidence for $M_1$.

To obtain the BIC for the direct and coded models, the model distributions are taken to be Gaussians whose means (for each item and delay) are given by the theoretical results of Equations 1 and 2, respectively, and whose variance is given by the empirically measured data variance across trials and subjects, computed separately per item and delay. We used the parameters $N=10,1/2𝒟=2.28$ for the well-coded storage model, and $(2𝒟/N)=3.24\times {10}^{-7}$ for the direct storage model, to obtain ${\displaystyle \mathrm{\Delta }\mathrm{B}\mathrm{I}\mathrm{C}=172.67}$. The empirical response variance is computed over each trial for each subjects, for a total of $n=660$ observations for each $(T,K)$ or (delay interval, item number) pair. The number of parameters is $k=1$ for direct storage and $k=2$ for well-coded storage. Setting the parameter numbers to $k=1+4$ and $k=2+4$ to take into account the 4 values of response errors at the shortest delay at $T=100$ ms does not change the ${\displaystyle \mathrm{\Delta }\mathrm{B}\mathrm{I}\mathrm{C}}$ score because the score is dominated by the likelihood term, so that these changes in the parameter penalty term have negligible effect.

Modeling short-term memory as direct storage of variables in persistent activity networks, produces results that are inconsistent with the data, as shown in the main paper. To obtain predictions for persistence and capacity through direct storage in persistent activity networks, first consider storing a single circular orientation variable, for a single bar in the delayed orientation matching task, as a bump in one ring network (Ben-Yishai et al., 1995; Amit, 1992; Zhang, 1996). The ring network would have neurons from all the $N$ storage networks in our short-term memory system pooled together, thus the network is $N$ times larger. The mean squared error of a variable stored in a continuous attractor neural network with stochastic neural spiking grows linearly with the storage interval $T$ over short intervals (with ‘short’ defined as all intervals before the root-mean squared error has grown to be an appreciable fraction of the range of the variable, $2\pi$). Let $\varphi /\mathrm{\Phi }$ be the coded variable, with $\varphi \in [0,\mathrm{\Phi }]$. If the rate of growth of error in the individual storage networks of the main paper is $2𝒟$ (recall that $\overline{D}=D/P$, where $D$ is coefficient of diffusion (Burak and Fiete, 2012); thus, the quantity $\overline{D}$ describes the rate at which the stored variable drifts away from its initial value, normalized by the squared range of the variable, per unit power of the representation; alternatively, we may think of the total representional power as being normalized to 1 in all cases), then the rate of growth of squared error in the single ring network is $2𝒟/N$ (Burak and Fiete, 2012). The factor of $N$ enters because if all other quantities are held fixed, the diffusion coefficient in continuous attractor memory networks is inversely proportional to network size. Thus, the squared error in the variable at short times $T$ is given by $⟨{(\varphi (T)-\varphi (0))}^2⟩/{\mathrm{\Phi }}^2=2𝒟T/N$. In other words, we have

Next, consider storing $K$ scalar variables, with each component ranging in $[0,\mathrm{\Phi }]$, and represented in one of $K$ different small networks, constructed from the single storage network above. Thus, its size is $1/K$ of the above. Relative to Equation 20 above, we therefore have

In other words, for memory systems involving direct storage in persistent activity networks without special encoding, we expect the squared error to grow linearly with $K$ and $T$. The prediction of uncoded storage in persistent activity networks can be compared directly with the prediction from encoded storage (Equation 2), because they involve the same parameters and the same resource use in the memory networks. While adding a proper encoding stage can reduce storage errors exponentially in $N$, uncoded storage results in decreases with $N$ that are merely polynomial (more specifically, scaling as $N^{-1}$).

Finally, one may consider directly storing the $K$-dimensional variable in a single persistent activity network that is a $K$-dimensional ring network (a $K$-torus). In this situation, the neurons have to be arranged so the number of neurons per linear dimension of the network scales as $N^{1/K}$. Thus, the rate of growth of squared error along each dimension of the network scales as $2𝒟/(N^{1/K})$, and we have

This scaling with $T$ remains linear, while the improvement in squared error with $N$ is weaker than the scaling in Equation 21 , which in turn is weaker than the scaling in Equation 2 , and consequently produces worse fits to the data than does Equation 21 . Therefore, we have chosen to contrast the better of two scenarios of direct (uncoded) storage, Equation 21 , against the predictions of the theory of short-term memory proposed in this work.

Here, we supply the data from individual subjects, as well as fits of the theory of Equation 2 and the direct storage model 1 to their performance.

The individual subject responses and the fits of the well-coded storage model are shown in Figure 4—figure supplement 1. We first plot the quality-of-fit or energy surface of the fits of the well-coded model to the individual subject data (top two rows in Figure 4—figure supplement 1), as the two parameters of the model are varied. These individual-subject solution spaces look qualitatively similar to the across-subject aggregates reported in the main manuscript. All subjects exhibit a 1D manifold of ‘good’ parameter settings, along which the model provides a reasonable match to the data. The quality of fit along the 1D manifold (valley) is shown in the next two rows of Figure 4—figure supplement 1; based on the local minima of these curves, we infer the optimal settings of $N$ and $1/2𝒟$ for each subject. The differences between individuals emerges in that the best $N$ values range between 2 and 20, and that for most subjects, the best values range between 4 and 11. Subjects with deviations in the optimal $N$ from this narrower range have essentially flat valleys between $N=2$ and $N=20$ (Figure 4—figure supplement 1), and thus the choice of $N$ is not strongly constrained.

The minimum fit errors are necessarily larger than the minimum fit errors for the across-subject averaged data, because of the higher variability of individual subject data (fewer trials per subject than total trials across subjects). Nevertheless, the normalized squared errors of the fits can be quite low, and the theory provides good fits to the psychophysics data for the individual subjects.

We also fit the individual subject data to the direct storage models, to be able to compare the predictions from the two models, Figure 4—figure supplement 2. We then compute the Bayesian Information Criterion score for both the direct storage model and the well-coded storage model, and report the $\mathrm{\Delta }BIC$ score for hypothesis comparison, Figure 4—figure supplement 2. Positive (negative) $\mathrm{\Delta }BIC$ scores indicate support for the well-coded (direct) storage model, and an absolute value of 10 or greater indicates very strong support. Note that the $\mathrm{\Delta }BIC$ scores for the individual subjects are much smaller in magnitude than the aggregate scores for all pooled data in the main manuscript, because the data set for individual subjects is smaller and has less statistical strength. Nevertheless, there is very strong support ($|\mathrm{\Delta }BIC|>10$) for the well-coded model in 4 out of 10 subjects, close to strong support for direct storage in 2 out of 10 subjects ($|\mathrm{\Delta }BIC|\approx 10$), positive support for direct storage in 2 subjects, and essentially insignificant support ($|\mathrm{\Delta }BIC|\approx 2$) in 2 remaining subjects.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Human subjects: The study reported here conform to the Declaration of Helsinki and all procedures were approved by the ethics committee of the National Hospital for Neurology and Neurosurgery (NHNN) prior to the study commencing. Research Ethics Committee number (ERC) 04/Q0406/60. Personal information about individuals was password protected and saved in compliance to the Data Protection Act 1998 (DPA).

© 2017, Koyluoglu et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.