A dynamic neural resource model bridges sensory and working memory

Reviewer #1 (Public Review):

Summary:
This study investigates how the neural representation of a stimulus transitions from that evoked by the presence of the stimulus (sensory) to one that exists only as a memory trace once the stimulus disappears (mnemonic). In simple terms, it explores the transition from so-called "iconic memory" (akin to residual sensory-driven neural activity) to working memory proper (self-sustained activity). The authors build a computational model for this transition and test it against data from two new psychophysical experiments plus two datasets from prior experiments.

Strengths and weaknesses:
I really liked this work. It considers a fairly complex process but builds a mechanistically comprehensive scheme that is intuitive and testable. This is a hefty paper; the full model built by the authors has a lot of moving parts. But these are all carefully justified, and in fact, many of them are specifically tested by fitting customized variants of the model to the experimental data (which are rich enough to distinguish all of these variants, not only quantitatively but also qualitatively). Said differently, both the assumptions used to build the model and the conclusions drawn after comparison with the experimental data are well justified. In the end, although it takes some effort to put the whole scheme together, I think the reader learns a lot about memory mechanisms. The Discussion is rich, as beyond working memory per se, the work relates to numerous issues (e.g., perception, attention, neural dynamics, population coding). Importantly, although part of the value of the study lies in the way it integrates many prior results into a cohesive framework, it also makes an important novel point: that iconic and working memory are not qualitatively different things, but rather just different extreme manifestations of the one, continuous process whereby perceptual information is stored (as a pattern of neural activation) and made accessible to other cognitive functions. In this conceptualization, working memory corresponds to a readout of activity a significant time (typically > 1 s) after stimulus offset, whereas iconic memory is consistent with a readout from the same neural population but immediately or very shortly after stimulus offset. This account not only is parsimonious but also provides a specific hypothesis (or a set of hypotheses) that can be tested further.

I did not find any major weaknesses. The paper does require some time and effort in order to appreciate all that it contains, but this is inevitable, as it aims to (1) build a compact but mechanistically detailed account of a process that is somewhat complex, and (2) test key predictions through psychophysical experiments that must be sufficiently rich. In the end, I found the effort quite rewarding.

Reviewer #2 (Public Review):

Summary:
Previous work has shown subjects can use a form of short-term sensory memory, known as 'iconic memory', to accurately remember stimuli over short periods of time (several hundred milliseconds). Working memory maintains representations for longer periods of time but is strictly limited in its capacity. While it has long been assumed that sensory information acts as the input to working memory, a process model of this transfer has been missing. To address this, Tomic and Bays present the Dynamic Neural Resource (DyNR) model. The DyNR model captures the dynamics of processing sensory stimuli, transferring that representation into working memory, the diffusion of representations within working memory, and the selection of memory for report.

The DyNR model captures many of the effects observed in behavior. Most importantly, psychophysical experiments found the greater the delay between stimulus presentation and the selection of an item from working memory, the greater the error. This effect also depended on working memory load. In the model, these effects are captured by the exponential decay of sensory representations (i.e., iconic memory) following the offset of the stimulus. Once the selection cue is presented, residual information in iconic memory can be integrated into working memory, improving the strength of the representation and reducing error. This selection process takes time, and is slower for larger memory loads, explaining the observation that memory seems to decay instantly. The authors compare the DyNR model to several variants, demonstrating the importance of the persistence of sensory information in iconic memory, normalization of representations with increasing memory load, and diffusion of memories over time.

Strengths:
The manuscript provides a convincing argument for the interaction of iconic memory and working memory, as captured by the DyNR model. The analyses are cutting-edge and the results are well captured by the DyNR model. In particular, a strength of the manuscript is the comparison of the DyNR model to several alternative variants.

The results provide a process model for interactions between iconic memory and working memory. This will be of interest to neuroscientists and psychologists studying working memory. I see this work as providing a foundation for understanding behavior in continuous working memory tasks, from either a mechanistic, neuroscience, perspective or as a high-water mark for comparison to other psychological process models.

Finally, the manuscript is well written. The DyNR model is nicely described and an intuition for the dynamics of the model is clearly shown in Figures 2 and 4.

Weaknesses:
Despite its strengths, the paper does have some (relatively minor) weaknesses. In particular, the authors could consider the role of sensory processing, and its limitations, and variability in selecting an item from working memory as other factors affecting memory accuracy.

Reviewer #3 (Public Review):

Summary:
The authors set out to formally contrast several theoretical models of working memory, being particularly interested in comparing the models regarding their ability to explain cueing effects at short cue durations. These benefits are traditionally attributed to the existence of a high capacity, rapidly decaying sensory storage which can be directly read out following short latency retro-cues. Based on the model fits, the authors alternatively suggest that cue-benefits arise from a freeing of working memory resources, which at short cue latencies can be utilized to encode additional sensory information into VWM.

A dynamic neural population model consisting of separate sensory and VWM populations was used to explain temporal VWM fidelity of human behavioral data collected during several working memory tasks. VWM fidelity was probed at several timepoints during encoding, while sensory information was available, and maintenance when sensory information was no longer available. Furthermore, set size and exposure durations were manipulated to disentangle contributions of sensory and visual working memory.

Overall, the model explained human memory fidelity well, accounting for set size, exposure time, retention time, error distributions, and swap errors. Crucially the model suggests that recall at short delays is due to post-cue integration of sensory information into VWM as opposed to direct readout from sensory memory. The authors formally address several alternative theories, demonstrating that models with reduced sensory persistence, direct readout from sensory memory, no set-size dependent delays in cue processing, and constant accumulation rate provide significantly worse fits to the data.

I congratulate the authors for this rigorous scientific work. I have only very few remarks that I hope the authors can clarify.