Working memory (WM), more than just passively maintaining inputs, is also an active process that modifies and even reorganizes information representations to guide future behavior (Baddeley, 2003). For instance, retro-cues during retention could enhance WM performance (Griffin and Nobre, 2003; Landman et al., 2003; Larocque et al., 2014) and modulate neural responses in different brain regions (Christophel et al., 2018; Yu et al., 2020), suggesting that attention could flexibly modulate information that has already been maintained in WM (Myers et al., 2017). In addition to top-down attention, contexts and structure in which the to-be-memorized items are embedded also influence memory performance (Brady et al., 2011; Brady and Tenenbaum, 2013; DuBrow and Davachi, 2013; Gershman et al., 2013; Jiang et al., 2000; Oberauer and Lin, 2017). A typical example is the serial position effect for sequence memory, that is the recently presented item shows better memory performance compared to early one (Broadbent and Broadbent, 1981; Burgess and Hitch, 1999; Gorgoraptis et al., 2011; Huang et al., 2018; Jones and Oberauer, 2013). However, it remains largely unknown how structure context modulates or reorganizes the way multiple WM items are represented and maintained in the human brain. Here, we particularly focused on sequence structure, an essential one that mediates many cognitive functions, for example sequence memory, speech, movement control (Davachi and DuBrow, 2015; Giraud and Poeppel, 2012; Polyn et al., 2009).

Previous neural recordings demonstrate that a sequence of items would elicit serial and temporally compressed reactivations, which might reflect a memory consolidation process that reorganizes the incoming inputs (Bahramisharif et al., 2018; Foster and Wilson, 2006; Huang et al., 2018; Kurth-Nelson et al., 2016; Liu et al., 2019; Siegel et al., 2009; Skaggs and McNaughton, 1996). Interestingly, recent modeling and empirical studies propose that, in addition to maintenance via sustained or serial reactivations (Curtis and D'Esposito, 2003; Goldman-Rakic, 1995; Vogel and Machizawa, 2004), information could also be stored in synaptic weights of the network, that is activity-silent view (Miller et al., 2018; Mongillo et al., 2008; Rose et al., 2016; Sprague et al., 2016; Stokes, 2015; Trübutschek et al., 2017; Wolff et al., 2017). In other words, multiple items could be maintained in latent or hidden states of the WM neural network.

How to access the WM information stored in the ‘activity-silent’ network? An impulse-response approach, by presenting a PING stimulus to transiently perturb the WM system, efficiently reactivates WM representations (Wolff et al., 2017). Moreover, attended but not unattended item is successfully reactivated, implying that top-down attention modulates the latent states WM items reside in Wolff et al., 2017; Wolff et al., 2020. Here, we used this method to assess whether a list of WM items of equal task relevance would be reorganized by imposed sequence structure in varied latent states and in turn show distinct reactivation profiles. The hypothesis is also motivated by our previous findings demonstrating backward reactivations for sequence memory, which implies a more excitable state for recent vs. early items (Huang et al., 2018).

We recorded magnetoencephalography (MEG) signals when human subjects (N = 24) retained a sequence of orientations and their ordinal positions. A time-resolved multivariate inverted encoding model (IEM) (Brouwer and Heeger, 2009; Brouwer and Heeger, 2011; Sprague et al., 2014) was used to reconstruct the neural representations of each orientation (i.e. 1st item, 2nd item) over time. Importantly, a nonspecific PING stimulus (i.e. impulse) was presented during retention, aiming to transiently perturb the WM network so that the latent states of the stored items could be accessed. Our results demonstrate a backward reactivation profile such that the impulse triggers the neural representation of the 2nd item first, followed by that of the 1st item, thus supporting their different latent states. Moreover, the neural reactivation pattern well predicts the recency effect in behavior. In contrast, in another MEG experiment when subjects (N = 24, new subjects) retained the same sequence without needing to memorize the ordinal structure, the two orientations show similar reactivation profiles. Finally, computational modeling demonstrates that the sequence contexts, instead of passive memory decay, largely characterizes the recency behavior. Our findings constitute converging evidence supporting a central function of sequence structure in WM via reorganizing multiple items in the brain (i.e. in varied latent states) and influencing memory behavior. Generally speaking, our findings provide new perspectives for the neural mechanism underlying task-related multi-item information storage in the WM system.

In two MEG experiments, we examined how the sequence structure imposed on a list of WM items shapes their neural representations in the human brain. Items located in different positions of a list are stored in distinct latent states of the WM neural network, being reactivated at different latencies. The reactivation pattern correlates with the recency effect in recognition behavior. In contrast, memorizing the same list of items without sequence structure requirements does not elicit the dissociated reactivations and displays weakened recency effect. Moreover, neural representations of WM items could neither be generalized from the encoding to reactivations nor across items during retention, further advocating the reorganization of items in WM network. Finally, a computational model on the behavioral data supports that the recency effect is mainly derived from abstract sequence structures rather than passive memory decay. Taken together, sequence information, as a form of abstract structure context, essentially modulates memory performance by reorganizing items into different latent states of the WM neural network.

It has long been posited that WM relies on sustained neuronal firing or activities (Curtis and D'Esposito, 2003; Goldman-Rakic, 1995; Vogel and Machizawa, 2004). Interestingly, recent neural recordings and computational modeling advocate that memory could also be maintained in synapse weights without necessarily relying on sustained activities (i.e. hidden state), through short-term neural plasticity (STP) principles (Miller et al., 2018; Mongillo et al., 2008; Rose et al., 2016; Sprague et al., 2016; Stokes, 2015; Trübutschek et al., 2017; Wolff et al., 2017). Recent studies, by developing an interesting impulse-response approach, show that the neural representation of task-relevant features could be successfully reactivated from an activity-silent network (Wolff et al., 2017; Wolff et al., 2020). Meanwhile, even during the presumably ‘activity-silent’ period, memory information is still represented in the alpha-band activities (Bocincova and Johnson, 2019; Fahrenfort et al., 2017; Sutterer et al., 2019). Our control analysis based on alpha-band activities during retention supports the view and reveals sustained reactivation profiles (prior to and after the PING stimulus) for both 1st and 2nd orientations (Figure 3—figure supplement 3). The active memory representation carried in alpha-band response tends to occur in parallel to the activity-silent storage, that is not disrupted or modified by the PING stimulus. Therefore, two types of mechanism – active, sustained representation and STP-based ‘activity-silent’ storage –operate together to mediate the working memory process, and future studies could examine their possible distinct functions.

Top-down attention modulates latent states that WM items reside in. For instance, the immediately task-relevant item is in a more excitable state and tends to be first reactivated, compared to ones that are potentially task-relevant in the future (e.g. Lewis-Peacock et al., 2012; Rose et al., 2016). Notably, WM items in the present study are equally task-relevant and have the same probability to be tested during recalling, and the results thus could not be explained in terms of attentional modulation. In addition to attentional modulation, low-level features could also modulate the excitability of neural populations. A theoretical model posits that multiple items, located in different excitable states according to their bottom-up saliency levels, would be reactivated at different phases within an alpha-band cycle (Jensen et al., 2012; Jensen et al., 2014). Here, given the serial presentation of items, the lately presented item would presumably have less memory decay and in turn higher saliency level, hence residing in a more excitable state. However, two aspects excluded the explanation. First, Experiment 2 used the same sequence of items as Experiment 1, yet did not reveal the backward profiles. Second, the computational modeling demonstrates that the passive memory decay could not reliably capture the recency effect; rather, it is the sequence structure that plays a central function in modulating recency effect.

Structure information has long been viewed to influence perception and memory, for example global precedence effect (Chen, 1982; Liu et al., 2017; Navon, 1977), reverse hierarchy theory (Ahissar and Hochstein, 2004). Recently, it has been suggested that structural and content can be encoded in an independent manner to facilitate memory generalization (Behrens et al., 2018; Bengio et al., 2013; Higgins et al., 2017). Memory performance also tends to be influenced by the contexts the items are embedded in DuBrow and Davachi, 2013; Fischer et al., 2020; Gershman et al., 2013; Harris, 1952; Mathias et al., 2020. In other words, structure that characterizes the relationship among WM items spontaneously reshapes their neural representations in WM. Our results show that memory representation is reorganized by task contexts, that is ordinal position in Experiment 1 vs. big and small labels in Experiment 2. Meanwhile, sequence structure seems to be a special one and reorganizes WM items into different latent states, which is not the case for other task-relevant dimension, for example big/small labels in Experiment 2. Thus, our results provide new converging behavioral, neural, and modeling evidence advocating the prominent influence of sequence structure on WM behavior and the underlying neural implementations – reorganization of items into different latent states.

The backward reactivation suggests that items located in the late ordinal position are stored in a more excitable state, compared to those in the early positions. The results are well consistent with our previous findings, whereby a temporal response function (TRF) approach was employed to tag the item-specific reactivations (Huang et al., 2018),yet indeed differ in important ways. First, the TRF latency refers to relative timing, whereas the present design assessed neural representations in absolute time after the PING, thus providing new evidence for the backward reactivation. Second, the modeling results and Experiment 2 exclude the passive memory decay accounts, an unresolved issue in previous study. In fact, reversed replay has been observed in many circumstances, for example during break after spatial experience (Foster and Wilson, 2006), performing a reasoning task (Kurth-Nelson et al., 2016). A recent study shows that during structure learning, a forward replay in spontaneous activities would reverse in direction when paired with reward (Liu et al., 2019), implying the involvement of reinforcement learning principles (Schultz et al., 1997; Sutton and Barto, 1998). Thus, a possible interpretation for the backward reactivation is that the item located in the late position of a list might serve as an anchoring point in memory for other items and is in turn maintained in a more excitable latent state, since the recent item receives less inference and is more reliable for memory retrieval.

Experiment 2 served as a control experiment to test a straightforward alternative explanation (passive memory decay) and has been carefully matched with Experiment 1 in many characteristics, for example same stimuli, WM task, task difficulty. Crucially, in order to make the two experiments comparable in task, subjects recalled orientations according to big/small label. However, the new task would potentially introduce a confounding strategy factor, that is, a direct comparison between the two orientations. Due to the big/small comparisons in Experiment 2, subject might not retain the two orientations precisely as Experiment 1 but just memorize their relative angle or verbal labels. This could possibly account for the less significant reactivations in Experiment2 as well as the comparable reactivations for the two orientations. However, several control analyses weakened the possibility. First, orientation information for both items is indeed precisely represented in terms of the task-relevant dimension (big/small labels) in Experiment 2 (Figure 4—figure supplement 1). Second, the behavioral results of Experiment2 are largely accounted for by the angular difference between the to-be-recalled orientation and probe but not by that between the two orientations, confirming that subjects retained two orientations rather than their relative angle in WM. Meanwhile, it remains unknown, at least from the present data, whether comparison task by itself introduced in Experiment 2 would account for the disruption of backward reactivations. Future studies employing a non-comparison task in sequence working memory while controlling other task loads is needed to address the important question.

We built a computational model (Bays et al., 2009) that incorporates passive memory decay and sequence structure aiming to understand the recency behavior. The model fitting separates the memory decay influence and confirms that it is the sequence context that mostly accounts for the recency effect. This is also consistent with a previous study revealing that low-level, absolute judgments fail to characterize the high-level, relative judgements (Ding et al., 2017). Another explanation of recency effect is the interference account, that is the final item in a list is free from the interference of subsequent items (Gorgoraptis et al., 2011), which awaits further studies to investigate. Although the current model only characterized the behavioral performance, the results are highly consistent with the MEG findings, that is decreased recency effect and comparable reactivations when sequence structure context was absent. Taken together, MEG recordings together with the computational model convergingly advocate an essential role of sequence context in WM and its neural implementation, that is reorganizing WM items into different latent states of the neural network.

Source data files are provided here: https://osf.io/9amq6/.

1. 1. Huang Q
   2. Zhang H
   3. Luo H

   (2021) Open Science Framework

   ID 9amq6. Sequence structure organizes items in varied latent states of working memory neural network.

   https://osf.io/9amq6

1. Conference

   1. Acerbi L
   2. Ma WJ

   (2017)

   Practical bayesian optimization for model fitting with bayesian adaptive direct search

   Advances in Neural Information Processing Systems. pp. 1836–1846.

   • Google Scholar
2. 1. Ahissar M
   2. Hochstein S

   (2004) The reverse hierarchy theory of visual perceptual learning

   Trends in Cognitive Sciences 8:457–464.

   https://doi.org/10.1016/j.tics.2004.08.011

   • PubMed
   • Google Scholar
3. 1. Baddeley A

   (2003) Working memory: looking back and looking forward

   Nature Reviews Neuroscience 4:829–839.

   https://doi.org/10.1038/nrn1201

   • PubMed
   • Google Scholar
4. 1. Bahramisharif A
   2. Jensen O
   3. Jacobs J
   4. Lisman J

   (2018) Serial representation of items during working memory maintenance at letter-selective cortical sites

   PLOS Biology 16:e2003805.

   https://doi.org/10.1371/journal.pbio.2003805

   • PubMed
   • Google Scholar
5. 1. Bays PM
   2. Catalao RF
   3. Husain M

   (2009) The precision of visual working memory is set by allocation of a shared resource

   Journal of Vision 9:7.

   https://doi.org/10.1167/9.10.7

   • PubMed
   • Google Scholar
6. 1. Behrens TEJ
   2. Muller TH
   3. Whittington JCR
   4. Mark S
   5. Baram AB
   6. Stachenfeld KL
   7. Kurth-Nelson Z

   (2018) What is a cognitive map? organizing knowledge for flexible behavior

   Neuron 100:490–509.

   https://doi.org/10.1016/j.neuron.2018.10.002

   • PubMed
   • Google Scholar
7. Conference

   1. Bengio Y
   2. Mesnil G
   3. Dauphin Y
   4. Rifai S

   (2013)

   Better mixing via deep representations

   International Conference on Machine Learning. pp. 552–560.

   • Google Scholar
8. 1. Bocincova A
   2. Johnson JS

   (2019) The time course of encoding and maintenance of task-relevant versus irrelevant object features in working memory

   Cortex 111:196–209.

   https://doi.org/10.1016/j.cortex.2018.10.013

   • PubMed
   • Google Scholar
9. 1. Brady TF
   2. Konkle T
   3. Alvarez GA

   (2011) A review of visual memory capacity: beyond individual items and toward structured representations

   Journal of Vision 11:4–34.

   https://doi.org/10.1167/11.5.4

   • Google Scholar
10. 1. Brady TF
    2. Tenenbaum JB

    (2013) A probabilistic model of visual working memory: incorporating higher order regularities into working memory capacity estimates

    Psychological Review 120:85–109.

    https://doi.org/10.1037/a0030779

    • PubMed
    • Google Scholar
11. 1. Broadbent DE
    2. Broadbent MHP

    (1981) Recency effects in visual memory

    The Quarterly Journal of Experimental Psychology Section A 33:1–15.

    https://doi.org/10.1080/14640748108400762

    • Google Scholar
12. 1. Brouwer GJ
    2. Heeger DJ

    (2009) Decoding and reconstructing color from responses in human visual cortex

    Journal of Neuroscience 29:13992–14003.

    https://doi.org/10.1523/JNEUROSCI.3577-09.2009

    • PubMed
    • Google Scholar
13. 1. Brouwer GJ
    2. Heeger DJ

    (2011) Cross-orientation suppression in human visual cortex

    Journal of Neurophysiology 106:2108–2119.

    https://doi.org/10.1152/jn.00540.2011

    • PubMed
    • Google Scholar
14. 1. Burgess N
    2. Hitch GJ

    (1999) Memory for serial order: a network model of the phonological loop and its timing

    Psychological Review 106:551–581.

    https://doi.org/10.1037/0033-295X.106.3.551

    • Google Scholar
15. 1. Chen L

    (1982) Topological structure in visual perception

    Science 218:699–700.

    https://doi.org/10.1126/science.7134969

    • PubMed
    • Google Scholar
16. 1. Christophel TB
    2. Iamshchinina P
    3. Yan C
    4. Allefeld C
    5. Haynes JD

    (2018) Cortical specialization for attended versus unattended working memory

    Nature Neuroscience 21:494–496.

    https://doi.org/10.1038/s41593-018-0094-4

    • PubMed
    • Google Scholar
17. 1. Curtis CE
    2. D'Esposito M

    (2003) Persistent activity in the prefrontal cortex during working memory

    Trends in Cognitive Sciences 7:415–423.

    https://doi.org/10.1016/S1364-6613(03)00197-9

    • PubMed
    • Google Scholar
18. 1. Davachi L
    2. DuBrow S

    (2015) How the Hippocampus preserves order: the role of prediction and context

    Trends in Cognitive Sciences 19:92–99.

    https://doi.org/10.1016/j.tics.2014.12.004

    • PubMed
    • Google Scholar
19. 1. Ding S
    2. Cueva CJ
    3. Tsodyks M
    4. Qian N

    (2017) Visual perception as retrospective bayesian decoding from high- to low-level features

    PNAS 114:E9115–E9124.

    https://doi.org/10.1073/pnas.1706906114

    • PubMed
    • Google Scholar
20. 1. DuBrow S
    2. Davachi L

    (2013) The influence of context boundaries on memory for the sequential order of events

    Journal of Experimental Psychology: General 142:1277–1286.

    https://doi.org/10.1037/a0034024

    • Google Scholar
21. 1. Ester EF
    2. Sprague TC
    3. Serences JT

    (2015) Parietal and frontal cortex encode Stimulus-Specific mnemonic representations during visual working memory

    Neuron 87:893–905.

    https://doi.org/10.1016/j.neuron.2015.07.013

    • PubMed
    • Google Scholar
22. Preprint

    1. Fahrenfort JJ
    2. Van Leeuwen J
    3. Foster JJ
    4. Awh E
    5. Olivers CNL

    (2017) Working memory implements distinct maintenance mechanisms depending on task goals

    bioRxiv.

    https://doi.org/10.1101/162537

    • Google Scholar
23. 1. Fischer C
    2. Czoschke S
    3. Peters B
    4. Rahm B
    5. Kaiser J
    6. Bledowski C

    (2020) Context information supports serial dependence of multiple visual objects across memory episodes

    Nature Communications 11:1–11.

    https://doi.org/10.1038/s41467-020-15874-w

    • Google Scholar
24. 1. Foster JJ
    2. Bsales EM
    3. Jaffe RJ
    4. Awh E

    (2017) Alpha-Band Activity Reveals Spontaneous Representations of Spatial Position in Visual Working Memory

    Current biology : CB 27:3216–3223.

    https://doi.org/10.1016/j.cub.2017.09.031

    • PubMed
    • Google Scholar
25. 1. Foster DJ
    2. Wilson MA

    (2006) Reverse replay of behavioural sequences in hippocampal place cells during the awake state

    Nature 440:680–683.

    https://doi.org/10.1038/nature04587

    • PubMed
    • Google Scholar
26. 1. Gershman SJ
    2. Schapiro AC
    3. Hupbach A
    4. Norman KA

    (2013) Neural context reinstatement predicts memory misattribution

    Journal of Neuroscience 33:8590–8595.

    https://doi.org/10.1523/JNEUROSCI.0096-13.2013

    • PubMed
    • Google Scholar
27. 1. Giraud AL
    2. Poeppel D

    (2012) Cortical oscillations and speech processing: emerging computational principles and operations

    Nature Neuroscience 15:511–517.

    https://doi.org/10.1038/nn.3063

    • PubMed
    • Google Scholar
28. 1. Goldman-Rakic PS

    (1995) Cellular basis of working memory

    Neuron 14:477–485.

    https://doi.org/10.1016/0896-6273(95)90304-6

    • PubMed
    • Google Scholar
29. 1. Gorgoraptis N
    2. Catalao RF
    3. Bays PM
    4. Husain M

    (2011) Dynamic updating of working memory resources for visual objects

    Journal of Neuroscience 31:8502–8511.

    https://doi.org/10.1523/JNEUROSCI.0208-11.2011

    • PubMed
    • Google Scholar
30. 1. Griffin IC
    2. Nobre AC

    (2003) Orienting attention to locations in internal representations

    Journal of Cognitive Neuroscience 15:1176–1194.

    https://doi.org/10.1162/089892903322598139

    • PubMed
    • Google Scholar
31. 1. Harris JD

    (1952) The decline of pitch discrimination with time

    Journal of Experimental Psychology 43:96–99.

    https://doi.org/10.1037/h0057373

    • PubMed
    • Google Scholar
32. 1. Higgins I
    2. Matthey L
    3. Pal A
    4. Burgess C
    5. Glorot X
    6. Botvinick M
    7. Mohamed S
    8. Lerchner A

    (2017)

    beta-VAE: learning basic visual concepts with a constrained variational framework

    Iclr 2:6.

    • Google Scholar
33. 1. Huang Q
    2. Jia J
    3. Han Q
    4. Luo H

    (2018) Fast-backward replay of sequentially memorized items in humans

    eLife 7:e35164.

    https://doi.org/10.7554/eLife.35164

    • PubMed
    • Google Scholar
34. 1. Jensen O
    2. Bonnefond M
    3. VanRullen R

    (2012) An oscillatory mechanism for prioritizing salient unattended stimuli

    Trends in Cognitive Sciences 16:200–206.

    https://doi.org/10.1016/j.tics.2012.03.002

    • PubMed
    • Google Scholar
35. 1. Jensen O
    2. Gips B
    3. Bergmann TO
    4. Bonnefond M

    (2014) Temporal coding organized by coupled alpha and gamma oscillations prioritize visual processing

    Trends in Neurosciences 37:357–369.

    https://doi.org/10.1016/j.tins.2014.04.001

    • PubMed
    • Google Scholar
36. 1. Jiang Y
    2. Olson IR
    3. Chun MM

    (2000) Organization of visual short-term memory

    Journal of Experimental Psychology: Learning, Memory, and Cognition 26:683–702.

    https://doi.org/10.1037/0278-7393.26.3.683

    • Google Scholar
37. 1. Jones T
    2. Oberauer K

    (2013) Serial-position effects for items and relations in short-term memory

    Memory 21:347–365.

    https://doi.org/10.1080/09658211.2012.726629

    • PubMed
    • Google Scholar
38. 1. Kok P
    2. Mostert P
    3. de Lange FP

    (2017) Prior expectations induce prestimulus sensory templates

    PNAS 114:10473–10478.

    https://doi.org/10.1073/pnas.1705652114

    • PubMed
    • Google Scholar
39. 1. Kurth-Nelson Z
    2. Economides M
    3. Dolan RJ
    4. Dayan P

    (2016) Fast sequences of Non-spatial state representations in humans

    Neuron 91:194–204.

    https://doi.org/10.1016/j.neuron.2016.05.028

    • PubMed
    • Google Scholar
40. 1. Landman R
    2. Spekreijse H
    3. Lamme VA

    (2003) Large capacity storage of integrated objects before change blindness

    Vision Research 43:149–164.

    https://doi.org/10.1016/S0042-6989(02)00402-9

    • PubMed
    • Google Scholar
41. 1. Larocque JJ
    2. Lewis-Peacock JA
    3. Postle BR

    (2014) Multiple neural states of representation in short-term memory? it's a matter of attention

    Frontiers in Human Neuroscience 8:5.

    https://doi.org/10.3389/fnhum.2014.00005

    • PubMed
    • Google Scholar
42. 1. Lewis-Peacock JA
    2. Drysdale AT
    3. Oberauer K
    4. Postle BR

    (2012) Neural evidence for a distinction between short-term memory and the focus of attention

    Journal of Cognitive Neuroscience 24:61–79.

    https://doi.org/10.1162/jocn_a_00140

    • PubMed
    • Google Scholar
43. 1. Liu L
    2. Wang F
    3. Zhou K
    4. Ding N
    5. Luo H

    (2017) Perceptual integration rapidly activates dorsal visual pathway to guide local processing in early visual Areas

    PLOS Biology 15:e2003646.

    https://doi.org/10.1371/journal.pbio.2003646

    • PubMed
    • Google Scholar
44. 1. Liu Y
    2. Dolan RJ
    3. Kurth-Nelson Z
    4. Behrens TEJ

    (2019) Human replay spontaneously reorganizes experience

    Cell 178:640–652.

    https://doi.org/10.1016/j.cell.2019.06.012

    • Google Scholar
45. 1. Maris E
    2. Oostenveld R

    (2007) Nonparametric statistical testing of EEG- and MEG-data

    Journal of Neuroscience Methods 164:177–190.

    https://doi.org/10.1016/j.jneumeth.2007.03.024

    • PubMed
    • Google Scholar
46. 1. Mathias SR
    2. Varghese L
    3. Micheyl C
    4. Shinn-Cunningham BG

    (2020) On the utility of perceptual anchors during pure-tone frequency discrimination

    The Journal of the Acoustical Society of America 147:371–380.

    https://doi.org/10.1121/10.0000584

    • PubMed
    • Google Scholar
47. 1. Miller EK
    2. Lundqvist M
    3. Bastos AM

    (2018) Working memory 2.0

    Neuron 100:463–475.

    https://doi.org/10.1016/j.neuron.2018.09.023

    • PubMed
    • Google Scholar
48. 1. Mongillo G
    2. Barak O
    3. Tsodyks M

    (2008) Synaptic theory of working memory

    Science 319:1543–1546.

    https://doi.org/10.1126/science.1150769

    • PubMed
    • Google Scholar
49. 1. Myers NE
    2. Rohenkohl G
    3. Wyart V
    4. Woolrich MW
    5. Nobre AC
    6. Stokes MG

    (2015) Testing sensory evidence against mnemonic templates

    eLife 4:e09000.

    https://doi.org/10.7554/eLife.09000

    • PubMed
    • Google Scholar
50. 1. Myers NE
    2. Stokes MG
    3. Nobre AC

    (2017) Prioritizing information during working memory: beyond sustained internal attention

    Trends in Cognitive Sciences 21:449–461.

    https://doi.org/10.1016/j.tics.2017.03.010

    • PubMed
    • Google Scholar
51. 1. Navon D

    (1977) Forest before trees: the precedence of global features in visual perception

    Cognitive Psychology 9:353–383.

    https://doi.org/10.1016/0010-0285(77)90012-3

    • Google Scholar
52. 1. Oberauer K
    2. Lin HY

    (2017) An interference model of visual working memory

    Psychological Review 124:21–59.

    https://doi.org/10.1037/rev0000044

    • PubMed
    • Google Scholar
53. 1. Oostenveld R
    2. Fries P
    3. Maris E
    4. Schoffelen JM

    (2011) FieldTrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data

    Computational Intelligence and Neuroscience 2011:1–9.

    https://doi.org/10.1155/2011/156869

    • PubMed
    • Google Scholar
54. 1. Polyn SM
    2. Norman KA
    3. Kahana MJ

    (2009) A context maintenance and retrieval model of organizational processes in free recall

    Psychological Review 116:129–156.

    https://doi.org/10.1037/a0014420

    • PubMed
    • Google Scholar
55. 1. Rose NS
    2. LaRocque JJ
    3. Riggall AC
    4. Gosseries O
    5. Starrett MJ
    6. Meyering EE
    7. Postle BR

    (2016) Reactivation of latent working memories with transcranial magnetic stimulation

    Science 354:1136–1139.

    https://doi.org/10.1126/science.aah7011

    • PubMed
    • Google Scholar
56. 1. Schultz W
    2. Dayan P
    3. Montague PR

    (1997) A neural substrate of prediction and reward

    Science 275:1593–1599.

    https://doi.org/10.1126/science.275.5306.1593

    • PubMed
    • Google Scholar
57. 1. Shin H
    2. Zou Q
    3. Ma WJ

    (2017) The effects of delay duration on visual working memory for orientation

    Journal of Vision 17:10.

    https://doi.org/10.1167/17.14.10

    • PubMed
    • Google Scholar
58. 1. Siegel M
    2. Warden MR
    3. Miller EK

    (2009) Phase-dependent neuronal coding of objects in short-term memory

    PNAS 106:21341–21346.

    https://doi.org/10.1073/pnas.0908193106

    • PubMed
    • Google Scholar
59. 1. Skaggs WE
    2. McNaughton BL

    (1996) Replay of neuronal firing sequences in rat Hippocampus during sleep following spatial experience

    Science 271:1870–1873.

    https://doi.org/10.1126/science.271.5257.1870

    • PubMed
    • Google Scholar
60. 1. Sprague TC
    2. Ester EF
    3. Serences JT

    (2014) Reconstructions of information in visual spatial working memory degrade with memory load

    Current Biology 24:2174–2180.

    https://doi.org/10.1016/j.cub.2014.07.066

    • PubMed
    • Google Scholar
61. 1. Sprague TC
    2. Ester EF
    3. Serences JT

    (2016) Restoring latent visual working memory representations in human cortex

    Neuron 91:694–707.

    https://doi.org/10.1016/j.neuron.2016.07.006

    • PubMed
    • Google Scholar
62. 1. Stokes MG

    (2015) 'Activity-silent' working memory in prefrontal cortex: a dynamic coding framework

    Trends in Cognitive Sciences 19:394–405.

    https://doi.org/10.1016/j.tics.2015.05.004

    • PubMed
    • Google Scholar
63. 1. Sutterer DW
    2. Foster JJ
    3. Adam KCS
    4. Vogel EK
    5. Awh E

    (2019) Item-specific delay activity demonstrates concurrent storage of multiple active neural representations in working memory

    PLOS Biology 17:e3000239.

    https://doi.org/10.1371/journal.pbio.3000239

    • PubMed
    • Google Scholar
64. Book

    1. Sutton RS
    2. Barto AG

    (1998)

    Introduction to Reinforcement Learning

    MIT press.

    • Google Scholar
65. 1. Taulu S
    2. Simola J

    (2006) Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements

    Physics in Medicine and Biology 51:1759–1768.

    https://doi.org/10.1088/0031-9155/51/7/008

    • PubMed
    • Google Scholar
66. 1. Trübutschek D
    2. Marti S
    3. Ojeda A
    4. King JR
    5. Mi Y
    6. Tsodyks M
    7. Dehaene S

    (2017) A theory of working memory without consciousness or sustained activity

    eLife 6:e23871.

    https://doi.org/10.7554/eLife.23871

    • PubMed
    • Google Scholar
67. 1. Vogel EK
    2. Machizawa MG

    (2004) Neural activity predicts individual differences in visual working memory capacity

    Nature 428:748–751.

    https://doi.org/10.1038/nature02447

    • PubMed
    • Google Scholar
68. 1. Wolff MJ
    2. Jochim J
    3. Akyürek EG
    4. Stokes MG

    (2017) Dynamic hidden states underlying working-memory-guided behavior

    Nature Neuroscience 20:864–871.

    https://doi.org/10.1038/nn.4546

    • PubMed
    • Google Scholar
69. 1. Wolff MJ
    2. Kandemir G
    3. Stokes MG
    4. Akyürek EG

    (2020) Unimodal and bimodal access to sensory working memories by auditory and visual impulses

    The Journal of Neuroscience 40:671–681.

    https://doi.org/10.1523/JNEUROSCI.1194-19.2019

    • PubMed
    • Google Scholar
70. 1. Yu Q
    2. Teng C
    3. Postle BR

    (2020) Different states of priority recruit different neural representations in visual working memory

    PLOS Biology 18:e3000769.

    https://doi.org/10.1371/journal.pbio.3000769

    • PubMed
    • Google Scholar

Acceptance summary:

Investigating the role that activity-silent memory plays in supporting sequence information is timely and highly important. The reported core finding is that sequence context organizes working memory items in distinct latent states which can be reactivated during retention. These are compelling findings providing novel insight into how multi-item information is maintained in working memory and how it depends on task context.

Decision letter after peer review:

Thank you for submitting your article "Sequence structure organizes items in varied latent states of working memory neural network" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Laura Colgin as the Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Lluís Fuentemilla (Reviewer #2).

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

1) Address the issue of whether the absence of orientation reactivations in Experiment 2 reflect a "reformatting" of the maintained memory representations based on the task demands that is unrelated to whether or not sequence information is stored?

2) Is it possible that orientation information in the present study actually was maintained in an active state, by a signal (e.g. in the alpha-band activity) that was not analysed? This could be investigated anchored in previous work demonstrating successful orientation decoding based on alpha-band activity.

3) The study would benefit from clarity on additional information on implementation on analysis. See below for specifics.

4) A better handling of the specifics of the comparative task presented in Experiment 2 would increase the relevance to other communities interested in how representations (sensory and in working memory) interact over time.

5) In previous work this group has used a metric called representation fidelity to show rhythmic fluctuations of representation of two attended orientations. Could a similar approach be applied here? It can potentially be done even before the ping to investigate for rhythmic fluctuations in the WM representations

Reviewer #1:

Huang et al., investigated the role that latent or activity-silent memory states play in supporting memory of sequence information by implementing an impulse response procedure to examine the reactivation profiles of sequentially presented orientations. Applying an inverted encoding model to patterns of MEG data revealed that the second orientation of a two-item sequence was reactivated from an activity-silent state before the first orientation in the sequence when observers were cued to report remembered orientations based on list position. Interestingly, the magnitude of reactivations observed for both items was related to the size of the recency effect observed across participants. In contrast, no difference in reactivation timing was observed when orientation magnitude rather than list position was used to cue the probed orientation. The manuscript concludes that sequence structure is maintained in working memory via the reorganization of information into distinct latent states and that this reorganization is an active process that occurs only when sequence information needs to be maintained.

Overall, the manuscript poses a timely and well-motivated question, and if the reported conclusions hold up to further scrutiny, they provide important insight into how sequences are maintained in working memory. More broadly these observations would also provide another line of evidence in favor of the emerging view that activity silent working memory is a distinct memory process from both active maintenance and long-term memory. The authors did an excellent job including control experiments and analyses as well as being up front about the limitations of their results and assumptions necessary to support their conclusions. Furthermore, the authors make a compelling case that temporal decay does not account for the observed recency effects.

Despite my enthusiasm about many aspects of the paper, the manuscript has several weaknesses that undermine my confidence in its primary conclusions. Specifically, I worry that the absence of orientation reactivations in Experiment 2 reflect a reformatting of the maintained memory representations based on the task demands that is unrelated to whether or not sequence information is stored. Additionally, I worry that orientation information in the present study was maintained in an active state, via a signal (alpha-band activity) that was not analyzed, and that the observed memory reactivations may reflect the contents of this signal rather than a distinct activity-silent signal. I've outlined these points in more detail below.

1. The primary conclusion of the paper is that maintaining order information results in a restructuring of latent memory representations that does not occur when maintenance of sequence information is unnecessary for completing the task. Support for this conclusion rests on the assumption that observers maintain orientation information throughout the entire delay in both experiments, while only the necessity of retaining sequence information differs between experiments. However, the task in Experiment 2 requires a mental comparison between both orientations in order to correctly identify which item in the sequence was a big or small orientation. This comparison process could be done a number of different ways. One way, that is in line with the main conclusion of the manuscript, is to maintain two overlapping orientations throughout the delay. If this strategy was employed, we would expect to see robust reactivations of both orientations after the ping, but no difference in reactivation latency. However, observers could also complete the task by reformatting each orientation representation into a more easily compared code such as a verbal label or point along the edge of the grating as each item is encoded. If this were the case, we would not expect to observe a reactivation of orientation information following the ping. The absence of a significant orientation reactivation for item 1 and observation of only a marginally significant reactivation for item 2 seems to provide stronger support for such a reformatting account than an order free orientation code in Experiment 2.

2. Recent work has shown that multivariate patterns of alpha-band activity (8 – 12 Hz) track centrally (Bocincova and Johnson, 2019; Fahrenfort, Leeuwen, Foster, Awh, and Olivers, 2017) and laterally (Fukuda, Kang, and Woodman, 2016; see Figure 9A) presented orientations as they are maintained in working memory. Thus, there is reason to assume that orientation information in the present study is actively represented throughout the delay-period via alpha-band activity. The primary conclusion of the manuscript is that latent or activity-silent memory states are essential for maintaining sequence information, thus it is important to establish whether or not orientation memory is supported by an active code throughout the delay. Now, it may be possible that a sustained code could operate in parallel with an activity silent code. However, to support the conclusions drawn in the manuscript it is critical to determine whether both codes exist and to provide evidence the observed reactivation effects could not be explained by disruption or reactivation of an active signal in response to the ping stimulus.

References:

Bocincova, A., and Johnson, J. S. (2019). The time course of encoding and maintenance of task-relevant versus irrelevant object features in working memory. Cortex, 111, 196-209.

Fahrenfort, J. J., Leeuwen, J. van, Foster, J., Awh, E., and Olivers, C. N. L. (2017). Working memory implements distinct maintenance mechanisms depending on task goals. BioRxiv, 162537.

Fukuda, K., Kang, M.-S., and Woodman, G. F. (2016). Distinct neural mechanisms for spatially lateralized and spatially global visual working memory representations. Journal of Neurophysiology, 116(4), 1715-1727.

Gorgoraptis, N., Catalao, R. F. G., Bays, P. M., and Husain, M. (2011). Dynamic Updating of Working Memory Resources for Visual Objects. Journal of Neuroscience, 31(23), 8502-8511.

Reviewer #2:

The current study investigated how the temporal structure of an encoded sequence of 2 items can be preserved during a short offline period of maintenance in working memory (WM) in humans. The work includes 3 different experimental sets of data and evidence from behavioral, MEG recordings and computational modelling. The results of the study revealed that WM preserves the temporal order of a just encoded sequence of visual two stimuli in "latent states" of neural activity. Latent states are thought to be an efficient way to store information in the brain but require inducing a transient perturbation to be identified empirically. The authors elicited this perturbation after encoding via the visual presentation of a PING stimulus and studied the whether the temporal pattern of reactivation of encoded items during the subsequent maintenance time period. In the first experiment, they showed that second encoded item reactivated before the first item during maintenance and that this reactivation pattern correlated to participants' accuracy in recalling the correct temporal order of the sequence in a subsequent test. To account for the possibility that the backward temporal reactivation profile could not be explained by varied decay strength to the PING stimulus at encoding, they run another experiment and showed that the backward reactivation structure of the sequence disappeared if participants were asked to maintain stimulus properties rather than in their temporal order dimension. Finally, computational modelling on a third data set supported the conclusion that temporal order maintenance was better estimated by parameters of sequential structure than passive memory decay. These data provide strong support to the conclusion that WM preserves the structure of a just encoded experience by reorganizing the neural activity into discrete and temporally separated latent states.

The conclusions of the study are well supported by the data, but some aspects of the data analysis need to be clarified and some mechanistic explanations may benefit from further investigation.

1) Most of the conclusions of the study rely on evidence derived from implementing a time-resolved multivariate analysis (MVPA) on MEG data. While this approach has been successfully used in previous studies with similar stimulation protocols, the study would benefit in clarity if they provided with additional information about its implementation. One such clarification would be to detail how MEG epochs were treated in the analysis. Because of the sequential structure of the encoding and maintenance it would be important to ensure that MVPA analysis preserved always the MEG signal at a specific trial level. It would be important to state if the MEG data was baseline corrected before implementing MVPA and if so, where this baseline period was set, given that maintenance period was preceded by PING stimulus which was in turn preceded by the presentation of the second item of the sequence. A detailed description of the number of trials included per condition and more details on the artifact rejection process seems necessary.

2) An important mechanism thought to support the temporal organization of sequenced items in WM is neural oscillations (i.e., theta and alpha-bands). In fact, the authors highlighted this possibility in the Discussion section, but they fail to explore this possibility on the data. I think the results of the current study would be strengthened if this mechanistic possibility was explored further. Several possible ways can address this issue in their data, given that MEG signal is well suited to this aim. One simple approach would be to investigate the dominance of spectral change modulations in specific frequency bands during the maintenance of temporal structure in experiment 1 but not during the maintenance of stimulus feature properties in experiment 2. One another possibility would be to assess for the existence of temporally structured memory reactivation during maintenance on the bases of MEG oscillatory activity at theta and/or alpha-band. I think that this type of analysis could increase the impact of the paper and extend the interest of the findings to a broader audience in the neuroscientific field.

3) Though the third control study results are in line with findings from experiment 1, they also reveal weaker effects on the 1 sec experimental condition, which is the one that coincides with in experiment 1. The third study however lost power as the number of participants dropped to N = 17 compared to the N = 24 in the first experiment. Given the relevance of the behavioral effects in experiment 1 for this study, I would recommend the authors to increase the sample size of experiment 3 to at least the same number as in experiment 1. That would help evaluate the consistency of the behavioral findings.

4) More details would help understand how the cluster-based permutation test was implemented in the data. How were permutations implemented? Was a cluster identified at the spatial or at the temporal scale?

5) In lines 486-487: " e.g., global precedence effect (Chen, 1982; L. Liu et al., 2017; Navon, 1977), reverse hierarchy theory (Ahissar and Hochstein, 2004), etc." I would recommend avoid using "etc" in the paper as it assumes the reader can make a correct guess about the other examples.

Reviewer #3:

This paper provides an innovative technique to the study of sustained activation and reactivation of representations in working memory. Whilst working memory representations have been investigated using the "ping" stimulation in combination with multivariate analyses of non-invasive physiology, this team is applying the same stimulation protocol in conjunction with an inverse encoding model (IEM), not investigated previously. Decoding performance is taken as evidence for reactivation (and from here on will be described as "reactivation markers" or "Activity").

These methodologies are combined with a set of behavioral experiments in order to tie decoding of working memory content to performance. Specifically, the reactivation patterns are used to account for a recency effect in a sequential working memory task.

In experiment 1 after a sequence of two items, a clear reactivation can be found first for the second stimulus of the sequence and later – for the first stimulus in the sequence. Differences in this degree of reactivation are correlated with the behavioral differences in performance between orientation discrimination for the first and second test grating. In this experiment, the items had to be retained for their orientation and order in the sequence. Later on, a probe stimulus would have to be compared to one of the two (cued by its ordinal position).

In a second experiment a near identical stimulation sequence was presented except now subjects were instructed to maintain both items for their angular distance to vertical. Later they were cued to one of the items based on this quantity – they were cued to perform either on the item that had a larger angular distance or the one that has a smaller angular distance. Thus, in addition to being indexed by angular distance to vertical, this task does include an additional element which the authors do not discuss explicitly. In order to select on a given trial which of the two items had a larger angular distance to vertical – a comparison between the items is required.

In this context, arguing that this protocol does not invoke sequence structure (e.g., "without a sequence structure imposed on them") might be overlooking the fact that not only the sequence isn't quite invoked, but it might be perturbed by the active comparison required.

The main findings, and most interesting element the authors provide in this paper is the linking of specific reactivation patterns and behavioral performance in Experiment 1. The methodologies there are solid and if regarded as a control experiment, Experiment 2 indeed points to the fact that task demands likely bring about a consistent reactivation pattern which in turn are related to the behavioral recency effect. The combination of the "ping" manipulation in order to probe latent states of the system is smart and neatly combined with the cutting edge multivariate analysis methods.

The interpretation of Experiment 2, however, are somewhat limited. The assumption of independence of representation for the different items in the sequence (layout above) ends up limiting the scope of analyses. Acknowledging the possibility that the two stimuli are simultaneously processed due to task demands (comparison) could inform interesting analyses that could better delineate the factors that affect working memory reactivation dynamics.

All in all the authors present an innovative combination of methodologies, and shed light on a putative mechanism for recency effects in visual working memory. The scope of the paper most definitely will be of interest to the community researching visual working memory both in human and in animal model. A better handling of the specifics of the "comparative" task presented in Experiment 2 might have increased the relevance to other communities interested in the way in which representations (sensory and in working memory) interact over time.

I would like to just elaborate on ways that could substantiate some of my claims above:

First – perhaps there are idiosynracies in the competitive interaction between wm representations in experiment 2 that require single trial analyses (behavior as well as physiology).

Second, in previous work this group has used a metric called representation fidelity to show rhythmic fluctuations in the representation of the two attended orientations even when coding was not significantly above chance. This could address my above points as well. It can potentially be done even before the ping to look for rhythmic fluctuations in the representation in WM.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Sequence structure organizes items in varied latent states of working memory neural network" for further consideration by eLife. Your revised article has been reviewed by 2 peer reviewers and the evaluation has been overseen by Laura Colgin as the Senior Editor, and a Reviewing Editor.

The manuscript has been improved and the additional analysis in the alpha-band has strengthened the results.

There are some remaining issues that need to be addressed. Reviewer 1 has provided a list of remaining concerns. We would like you to address all these concerns of which the overarching issues are:

1) Improve the Methods section in particular on how the alpha-band analysis and how the permutation tests were formed.

2) Some of the claims must be adjusted accordingly to the outcome of the statistical test following the correction for multiple comparisons.

3) The claims on experiment 2 should be moderated in light of the limitations of the experiment (see points 1 and 2).

Reviewer #1:

I appreciate the authors' responsiveness to the first round of reviews, and it's clear they put a lot of hard work into running additional analyses and revising the manuscript.

While I think the manuscript is improved as a result, I remain unconvinced that there is unambiguous support for the conclusion that sequence context, rather than task demands more generally, reorganizes the structure of latent memory states. Additionally, a number of the new analyses included in the revision and some of the previously included analyses lack a comprehensive description in the method section, and these missing details make it impossible to evaluate some of the new results that were included in the revision. I've outlined these comments along with others in more detail below:

1. I think the additional analyses included in the revision convincingly rule out the alternative explanation that I raised in my initial review, which is that observers recoded the format of orientation stimuli in Experiment 2 and that such a recoding drives the difference in results across experiments.

2. However, I'm not convinced that these additional analyses rule out the point raised by Reviewer 3, which is that the comparison process that was necessary to complete the task in Experiment 2 was not necessary in Experiment 1. Therefore, it's possible that engaging in this comparison process (even if it doesn't reformat the representations) eliminates the backwards reactivations observed in Experiment 1 and that the backwards reactivation observed in Experiment 1 would occur any time two stimuli are observed in a sequence as long as they don't need to be directly compared. Thus, I'm not convinced that Experiment 2 supports the strong conclusion drawn in the manuscript that the need to store sequence information itself creates a unique latent state for sequence memory. However, I do agree with the authors that the reported work provides support for the more open-ended conclusion that different latent state representations are recruited based on task demands. I think revising the manuscript to emphasize this more modest claim throughout or providing a clear explanation that this limitation of Experiment 2 remains unresolved in the discussion would be an acceptable solution.

3. No details are provided in the method section of the revised manuscript about the alpha analyses that were included in the revision. How was the filtering done? Was the analysis run on total alpha-power as in most previous decoding work or power relative to some sort of baseline (i.e., percent change or DB2). What was the rationale behind using alpha-band power relative to baseline used to identify significant sensors before conducting the multivariate analysis?

4. How was the training and testing conducted for the newly added big and small label analyses? Were all trial types used to train the model and then big and small label orientations were tested separately? If not, using a common training set and then testing on each analysis set separately might improve the power of the analysis (see Sprague et al., 2018 for more details on the benefits of using a "fixed" encoding model). I might be missing something, but it seems like the orientation bin with the smallest angular distance from vertical would never have been included as a "big orientation" trial and the orientation bin with the largest angular distance from vertical would never be included for the "small orientation" decoding is that right? If so, please describe this in the manuscript, and along with how the IEM procedure was adapted to account for these missing orientations.

5. The description of the cluster-based permutation test lines 633 – 639 is still difficult to follow and each step and decision needs to be laid out more clearly. It might help to cite a paper (or papers) that used the same approach applied here and to then report the thresholds set for the current manuscript. For example, some parts of the description sound like you used the approach applied by Sutterer et al. 2019 to identify clusters of above chance orientation selectivity and calculated a null distribution by first shuffling the orientation bin labels and re-running the IEM, repeating this process 1000 times and recording the largest cluster of above chance selectivity observed in each permutation. However, the current description of the analysis states that condition labels, rather than trial labels were shuffled. This approach would enable the identification of clusters where the selectivity differed between two conditions (e.g., big > small orientations), but it's not clear to me how this approach would allow for the identification of clusters of above chance selectivity within a single condition (i.e., when was orientation selectivity for big items higher than expected by chance). Finally, there should be some description of how the permutation test was adjusted for the addition and subtraction analyses. For instance, to calculate the null distribution for the additive analyses that were performed, permuted slope values for item 1 and item 2 (or the big and small orientations) should have been summed before calculating the permuted t-values, were they?

6. The manuscript and response letter refer to effects that were greater than.05 and/or that did not survive multiple comparisons corrections as significant. I know it's frustrating that some of these slopes look "close to significant" or "would have been significant without multiple comparisons correction", but there is no such thing as close to significant in frequentist statistics, so a cluster corrected p value of >.05 should not be reported as significant. The revised manuscript should be adjusted accordingly and arguments resting on the fact that non-significant time courses "look similar" should be removed. Below are a few instances of this that I noticed, but there may be more throughout the manuscript.

a. line 195 – 196 the reactivation of the first item was not significant following cluster correction. This isn't a huge problem since other analyses confirm the relationship between the reactivation of item one and behavioral performance, but it should be reported correctly.

b. Line 258, Decoding performance was not above chance after correcting for multiple comparisons for either item even after changing the permutation test from the two-tailed test used in Experiment 1 to a one-tailed test. Thus, it isn't informative to draw conclusions about whether the reactivation profile of the two items is similar when there is no evidence that each individual item was reactivated in the first place.

c. Line 266 only one of these windows is significant.

7. Line 369- 373: It would be nice to see more of the reasoning that the authors included in the response statement to explain to readers why the existence of this parallel active code does not pose a problem for the activity silent account.

8. Line- 440 (data not shown): show the behavioral analysis as a supplemental figure or report the result in line if it's used to provide support an argument.

References:

Sprague, T. C., Adam, K. C. S., Foster, J. J., Rahmati, M., Sutterer, D. W., and Vo, V. A. (2018). Inverted Encoding Models Assay Population-Level Stimulus Representations, Not Single-Unit Neural Tuning. Eneuro, 5(3), ENEURO.0098-18.2018.

Sutterer, D. W., Foster, J. J., Adam, K. C. S., Vogel, E. K., and Awh, E. (2019). Item-specific delay activity demonstrates concurrent storage of multiple active neural representations in working memory. PLoS Biology, 17(4), e3000239.

Reviewer #2:

The authors did a great job and successfully addressed all my previous concerns. Specifically, I think the addition of the results at the alpha-band clearly strengthens the previous findings and makes the paper more appealing to a broader audience.

Essential revisions:

1) Address the issue of whether the absence of orientation reactivations in Experiment 2 reflect a "reformatting" of the maintained memory representations based on the task demands that is unrelated to whether or not sequence information is stored?

We thank the reviewer for raising the important concern, that is, whether the weak reactivation of orientation information in Experiment 2 was due to a “reformatting” of the memory representations given different task demands (making big/small comparison between two orientations in Experiment 2). Specifically, the reviewer worried that “observers could also complete the task by reformatting each orientation representation into a more easily compared code such as a verbal label or point along the edge of the grating as each item is encoded. If this were the case, we would not expect to observe a reactivation of orientation information following the ping”.

This is a very interesting hypothesis, and we believe that several aspects as well as the new analysis results exclude this interpretation. First, we acknowledge that the reactivation for second orientation by itself in Experiment 2 was not significant, but it is noteworthy that the second item still showed a similar trend as that for item 1, and their summed reactivation displayed significant reactivations after PING (Figure 4I), thus to some extent supporting their similar reactivation profiles.

Second and most importantly, we have performed a new analysis to test the possibility raised by the reviewer. Specifically, instead of decoding orientation information based on ordinal position (first or second) as we originally did, we decoded orientation information based on “bigger/smaller” label in Experiment 2. As shown in Figure 4—figure supplement 1B, the Big- and Small-labeled orientations showed significant reactivations with similar temporal profiles after PING (Figure 4—figure supplement 1B; big orientation: 0.14 – 0.23 s, cluster p = 0.042; small orientation: 0.08 – 0.16 s, p < 0.05, uncorrected), and the summation results further support their comparable reactivation patterns (Figure 4—figure supplement 1C; 0.1 – 0.2 s, cluster p = 0.030). Therefore, the orientation information attached to the two labels (Big and Small) is indeed precisely represented in Experiment 2, rather than being encoded as a verbal label or point along a continuum dimension, as worried by the reviewer.

Moreover, we also performed the big/small-labelled decoding analysis during the encoding period. As shown in Figure 4—figure supplement 1A, the two orientations showed overlapping profiles locked to the presented stimuli, which makes sense since the big and small-labeled orientations occurred equally at the two positions. As a control, we also performed the same analysis for Experiment 1. As shown in Figure 4—figure supplement 1, successful decoding was observed during the encoding period (Figure 4—figure supplement 1D), but not during retention (Figure 4—figure supplement 1E, F). Therefore, task demands modulate how the memory system reorganizes orientation representations in the brain, i.e., big/small-labeled orientation decoding in Exp2 (task-relevant) but not in Exp1 (task-irrelevant).

Taken together, the weak orientation reactivations observed in Experiment 2 in the original results are not likely due to a reformatting of memory representation in verbal labels or point along a continuum dimension. Rather, orientation information is still represented precisely but reorganized according to big/small labels, which could be regarded as a type of task-dependent ‘reformatting’.

The new results have been added in Figure 4—figure supplement 1 and Discussion (Page 14-16).

2) Is it possible that orientation information in the present study actually was maintained in an active state, by a signal (e.g. in the alpha-band activity) that was not analysed? This could be investigated anchored in previous work demonstrating successful orientation decoding based on alpha-band activity.

We thank the reviewer for the insightful suggestion. We have performed new analysis on the alpha-band (8-12 Hz) activities throughout the maintaining period to test whether there are parallel active memory representations during retention.

First, we found significant alpha-band activation clusters (compared to baseline) in posterior sensors, during the maintaining period (Author response image 1A, B). We then performed the decoding analysis on the alpha-band power of parietal and occipital sensors (150 sensors in total) throughout the maintaining period. As shown below, both the first and second orientations could be successfully decoded throughout retention in Experiment 1 (Figure 3—figure supplement 3A; see Figure 3—figure supplement 3B for their summed results), thus consistent with previous findings (Bocincova and Johnson, 2019; Fahrenfort, et al., 2017; Fukuda, et al., 2016). Furthermore, the decoding performances showed a relatively sustained pattern (before and after PING stimulus), suggesting that the ‘active memory representation’ carried in alpha-band occurs in parallel to the activity-silent storage and is not disrupted or modified by the PING stimulus.

Furthermore, we did the same alpha-band decoding analysis in Experiment 2, based on big/small labels. Similarly, as shown in Figure 3—figure supplement 3C, D, the orientations could also be successfully decoded in the alpha-band signals and displayed a sustained profile.

We are very grateful for the reviewer’s suggestion which has essentially expanded our findings and advances our understanding about the neural mechanism for sequence working memory. We have added the new results in Figure 3—figure supplement 3 as well as in Discussion (Page 13).

Author response image 1

Download asset Open asset

3) The study would benefit from clarity on additional information on implementation on analysis. See below for specifics

We are sorry for the unclear specifications. We have added more details (as below) in Methods now (Page 19-23).

1. The MEG data was first baseline-corrected before decoding analysis. Specifically, the time range from -500 ms to 0 ms relative to the onset of first item in each trial was used as baseline to be subtracted.

2. Each participant in Experiment 1 and Experiment 2 should complete 280 trials in total. The first and the second grating were chosen from 7 orientation (25° –175° in 25° step), and each orientation occurred 40 times with random order. In each trial, the two orientations were drawn independently, but with a constraint that they should at least differ by 25°. The angular differences between a memory item and the corresponding probe were uniformly distributed across seven angle differences (± 3°, ± 6°, ± 9°, ± 13°, ± 18 °, ± 24°, ± 30°). Each angle difference had 20 trials.

3. To identify artifacts, the variance (collapsed over channels and time) was first calculated for each trial. Trials with excessive variances were removed. Next, to ensure that each orientation had the same number of trials, we set the minimum number of trials per orientation for decoding analysis to be 37 in each subject, which resulted in at least 259 trials per subject.

4. About the cluster-based permutation test, we first averaged the decoding performance over all trials belonging to each of the experimental conditions. The test statistics was calculated (t-test) at each time point based on which the significant clusters were found. Cluster-level statistics was calculated by computing the size of cluster. Next, a Monte-Carlo randomization procedure was used (randomizing data across conditions for 1000 times) to estimate the significance probabilities for each cluster.

4) A better handling of the specifics of the comparative task presented in Experiment 2 would increase the relevance to other communities interested in how representations (sensory and in working memory) interact over time.

Thanks for the insightful suggestion. Below we discussed the specifics of the comparative task in Experiment 2.

As acknowledged by the reviewers, Experiment 2 served as a control experiment to test a straightforward alternative explanation, that is, whether the observation is solely due to passive memory decay that arises from sequential presentation of the two orientations. We have carefully matched Experiment 1 and Experiment 2 in many characteristics, e.g., same stimuli, WM task, task difficulty (both around 75%). Crucially, in order to make the two experiments comparable in task demands as close as possible, subjects were instructed to recall orientation information according to big/small label instead of first and second label.

The reviewers raised an interesting concern about the additional comparison task embedded in Experiment 2, i.e., subjects might make a direct comparison between the two orientations rather than memorizing two orientations to achieve the task, which would potentially account for the observations (comparable reactivation profiles).

To address the concern, we fitted a generalized linear mixed-effects model to behavioral performance in Experiment 2, with the angular difference between first and second memory items and that between the to-be-retrieved memory item and probe item as independent variables. We found that only the angular difference between to-be-retrieved memory item and probe accounts for the behavioral performance (β = 0.0013, t = 3.45, p < 0.001), whereas the orientation between memory items could not (β < 0.0001, t = 0.50, p = 0.62). The results suggest that the memory performance in Experiment 2 was not simply mediated by the direct comparison between orientations. Instead, subjects retained two orientations in memory and retrieved the corresponding one and compared it to the probe. The possible reason that subjects did not use a comparison strategy might be the simple task employed here, i.e., relative large angular difference between the first and first orientations. Future studies using more difficult task could address the possibility and potentially reveal different neural reactivation patterns.

Finally, as mentioned in response to the first point, we have performed a new decoding analysis based on big/small labels and revealed significant reactivation for both first and second orientations in Experiment 2 but not in Experiment 1. This supports that task demands indeed modulate how the memory system reorganizes information representations in the brain, a type of task-dependent ‘reformatting’ of sensory information in working memory.

We have added more discussions about the comparison concern in Experiment 2 in Discussion (Page 14-16).

5) In previous work this group has used a metric called representation fidelity to show rhythmic fluctuations of representation of two attended orientations. Could a similar approach be applied here? It can potentially be done even before the ping to investigate for rhythmic fluctuations in the WM representations

We thank the reviewer for the suggestion and have now calculated the representation fidelity as used in our previous study (Mo et al., 2019). As shown in Author response image 2, the fidelity results were actually very similar to the slope index results, during both the encoding and maintaining periods.

Author response image 2

Download asset Open asset

According to the reviewer’s suggestion, we further examined possible rhythmic fluctuations in WM representations during retention. Since the raw results did not show rhythmic profiles (Author response image 2B, D), we calculated the second-to-first cross-correlation coefficient to examine the temporal relationship between the first and second orientation reactivations. Note that rhythmic profiles would predict correlations at periodic time lags. We first performed the cross-correlation analysis for temporal periods before PING (from the disappearance of the second item to the onset of the PING). As in Author response image 3A, C, the time lag between the first and second item was around 0 for both Experiment 1 and 2, suggesting that the two items showed similar temporal profiles. Next, we did the cross-correlation analysis for periods after PING. Interestingly, Experiment 1 showed cross-correlation around 300 ms (Author response image 3B) while Experiment 2 showed correlation around 0 (Author response image 3D), consistent with the observed sequential reactivations.

Taken together, the results do not reveal rhythmic fluctuations between WM items during retention, suggesting that WM system, especially in terms of the activity-silent state, does not entail rhythmic competition between multiple items, which indeed occurs when multiple features are physically presented and rival for attention (Mo et al., 2019).

Author response image 3

Download asset Open asset

Reviewer #1:

[…] Despite my enthusiasm about many aspects of the paper, the manuscript has several weaknesses that undermine my confidence in its primary conclusions. Specifically, I worry that the absence of orientation reactivations in Experiment 2 reflect a reformatting of the maintained memory representations based on the task demands that is unrelated to whether or not sequence information is stored. Additionally, I worry that orientation information in the present study was maintained in an active state, via a signal (alpha-band activity) that was not analyzed, and that the observed memory reactivations may reflect the contents of this signal rather than a distinct activity-silent signal. I've outlined these points in more detail below.

1. The primary conclusion of the paper is that maintaining order information results in a restructuring of latent memory representations that does not occur when maintenance of sequence information is unnecessary for completing the task. Support for this conclusion rests on the assumption that observers maintain orientation information throughout the entire delay in both experiments, while only the necessity of retaining sequence information differs between experiments. However, the task in Experiment 2 requires a mental comparison between both orientations in order to correctly identify which item in the sequence was a big or small orientation. This comparison process could be done a number of different ways. One way, that is in line with the main conclusion of the manuscript, is to maintain two overlapping orientations throughout the delay. If this strategy was employed, we would expect to see robust reactivations of both orientations after the ping, but no difference in reactivation latency. However, observers could also complete the task by reformatting each orientation representation into a more easily compared code such as a verbal label or point along the edge of the grating as each item is encoded. If this were the case, we would not expect to observe a reactivation of orientation information following the ping. The absence of a significant orientation reactivation for item 1 and observation of only a marginally significant reactivation for item 2 seems to provide stronger support for such a reformatting account than an order free orientation code in Experiment 2.

Please see our response to the first point in Essential Revision.

2. Recent work has shown that multivariate patterns of alpha-band activity (8 – 12 Hz) track centrally (Bocincova and Johnson, 2019; Fahrenfort, Leeuwen, Foster, Awh, and Olivers, 2017) and laterally (Fukuda, Kang, and Woodman, 2016; see Figure 9A) presented orientations as they are maintained in working memory. Thus, there is reason to assume that orientation information in the present study is actively represented throughout the delay-period via alpha-band activity. The primary conclusion of the manuscript is that latent or activity-silent memory states are essential for maintaining sequence information, thus it is important to establish whether or not orientation memory is supported by an active code throughout the delay. Now, it may be possible that a sustained code could operate in parallel with an activity silent code. However, to support the conclusions drawn in the manuscript it is critical to determine whether both codes exist and to provide evidence the observed reactivation effects could not be explained by disruption or reactivation of an active signal in response to the ping stimulus.

References:

Bocincova, A., and Johnson, J. S. (2019). The time course of encoding and maintenance of task-relevant versus irrelevant object features in working memory. Cortex, 111, 196-209.

Fahrenfort, J. J., Leeuwen, J. van, Foster, J., Awh, E., and Olivers, C. N. L. (2017). Working memory implements distinct maintenance mechanisms depending on task goals. BioRxiv, 162537.

Fukuda, K., Kang, M.-S., and Woodman, G. F. (2016). Distinct neural mechanisms for spatially lateralized and spatially global visual working memory representations. Journal of Neurophysiology, 116(4), 1715-1727.

Gorgoraptis, N., Catalao, R. F. G., Bays, P. M., and Husain, M. (2011). Dynamic Updating of Working Memory Resources for Visual Objects. Journal of Neuroscience, 31(23), 8502-8511.

Please see our response to the second point in Essential Revision.

Reviewer #2:

[…] The conclusions of the study are well supported by the data, but some aspects of the data analysis need to be clarified and some mechanistic explanations may benefit from further investigation.

1) Most of the conclusions of the study rely on evidence derived from implementing a time-resolved multivariate analysis (MVPA) on MEG data. While this approach has been successfully used in previous studies with similar stimulation protocols, the study would benefit in clarity if they provided with additional information about its implementation. One such clarification would be to detail how MEG epochs were treated in the analysis. Because of the sequential structure of the encoding and maintenance it would be important to ensure that MVPA analysis preserved always the MEG signal at a specific trial level. It would be important to state if the MEG data was baseline corrected before implementing MVPA and if so, where this baseline period was set, given that maintenance period was preceded by PING stimulus which was in turn preceded by the presentation of the second item of the sequence. A detailed description of the number of trials included per condition and more details on the artifact rejection process seems necessary.

Sorry for previous unclear specifications, and please see our response to the 3^nd point in Essential Revision.

2) An important mechanism thought to support the temporal organization of sequenced items in WM is neural oscillations (i.e., theta and alpha-bands). In fact, the authors highlighted this possibility in the Discussion section, but they fail to explore this possibility on the data. I think the results of the current study would be strengthened if this mechanistic possibility was explored further. Several possible ways can address this issue in their data, given that MEG signal is well suited to this aim. One simple approach would be to investigate the dominance of spectral change modulations in specific frequency bands during the maintenance of temporal structure in experiment 1 but not during the maintenance of stimulus feature properties in experiment 2. One another possibility would be to assess for the existence of temporally structured memory reactivation during maintenance on the bases of MEG oscillatory activity at theta and/or alpha-band. I think that this type of analysis could increase the impact of the paper and extend the interest of the findings to a broader audience in the neuroscientific field.

Thank you for the insightful suggestion and we have performed new analysis.

Please see our response to the second point in Essential Revision.

3) Though the third control study results are in line with findings from experiment 1, they also reveal weaker effects on the 1 sec experimental condition, which is the one that coincides with in experiment 1. The third study however lost power as the number of participants dropped to N = 17 compared to the N = 24 in the first experiment. Given the relevance of the behavioral effects in experiment 1 for this study, I would recommend the authors to increase the sample size of experiment 3 to at least the same number as in experiment 1. That would help evaluate the consistency of the behavioral findings.

We have increased the sample size of Experiment 3 to 24, which showed the consistent results as before. The results and Figure 5 have been updated (Page 10-12, 18).

4) More details would help understand how the cluster-based permutation test was implemented in the data. How were permutations implemented? Was a cluster identified at the spatial or at the temporal scale?

Sorry for the unclear description. We first averaged the decoding performance over all trials belonging to each of the experimental conditions. The test statistics was calculated (t-test) at each time point based on which the significant clusters were found. Cluster-level statistics was calculated by computing the size of cluster. Next, a Monte-Carlo randomization procedure was used (randomizing data across conditions for 1000 times) to estimate the significance probabilities for each cluster. Thus, a cluster was identified at temporal scale.

Details have been added in Methods (Page 22-23).

5) In lines 486-487: " e.g., global precedence effect (Chen, 1982; L. Liu et al., 2017; Navon, 1977), reverse hierarchy theory (Ahissar and Hochstein, 2004), etc." I would recommend avoid using "etc" in the paper as it assumes the reader can make a correct guess about the other examples.

Corrected.

Reviewer #3:

[…] The interpretation of Experiment 2, however, are somewhat limited. The assumption of independence of representation for the different items in the sequence (layout above) ends up limiting the scope of analyses. Acknowledging the possibility that the two stimuli are simultaneously processed due to task demands (comparison) could inform interesting analyses that could better delineate the factors that affect working memory reactivation dynamics.

We thank the reviewer for the insightful suggestion. We have performed a new decoding analysis based on big/small labels (task-relevant) and revealed significant reactivation for both the first and second orientations in Experiment 2 but not in Experiment 1. The results supports the reviewer’s view, suggesting that task demands indeed modulate how the memory system reorganizes information representations in the brain, a type of task-dependent ‘reformatting’ of sensory information in working memory.

Please also see more details in our response to the first point in Essential Revision.

All in all the authors present an innovative combination of methodologies, and shed light on a putative mechanism for recency effects in visual working memory. The scope of the paper most definitely will be of interest to the community researching visual working memory both in human and in animal model. A better handling of the specifics of the "comparative" task presented in Experiment 2 might have increased the relevance to other communities interested in the way in which representations (sensory and in working memory) interact over time.

Please see our response to the 4^th point in Essential Revision.

Comments for the authors:

I would like to just elaborate on ways that could substantiate some of my claims above:

First – perhaps there are idiosynracies in the competitive interaction between wm representations in experiment 2 that require single trial analyses (behavior as well as physiology).

Thank you for raising the concern. Following this suggestion, we have calculated the first-to-second cross-correlation coefficient in each trial and plotted the distribution of the time lag with the highest correlation coefficient across trials and participants. It is clear that the peak centers around 0, suggesting the two items’ comparable reactivations at single trial level.

Author response image 4

Download asset Open asset

Moreover, since participants recalled one item per trial, we could not examine their possible competition relationship in single trial. We thus examined whether the behavioral performance in Experiment 2 could be accounted for by the angular distance between memory items, by fitting a generalized linear mixed-effects model (Matlab function: fitglme). We found that only the angular difference between the target memory item and probe item accounts for the behavioral performance (β = 0.0013, t = 3.45, p < 0.001), while angular difference between two memory items could not (β < 0.0001, t = 0.50, p = 0.62). This result further supports that memory performance in Experiment 2 does not derive from competition between memory items.Taken together, the observed comparable reactivations for the two orientations is not likely due to the average of a competition profile in single trial.

Second, in previous work this group has used a metric called representation fidelity to show rhythmic fluctuations in the representation of the two attended orientations even when coding was not significantly above chance. This could address my above points as well. It can potentially be done even before the ping to look for rhythmic fluctuations in the representation in WM.

Please see our response to the fifth point in Essential Revision.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Sequence structure organizes items in varied latent states of working memory neural network" for further consideration by eLife. Your revised article has been reviewed by 2 peer reviewers and the evaluation has been overseen by Laura Colgin as the Senior Editor, and a Reviewing Editor.

The manuscript has been improved and the additional analysis in the alpha-band has strengthened the results.

There are some remaining issues that need to be addressed. Reviewer 1 has provided a list of remaining concerns. We would like you to address all these concerns of which the overarching issues are:

1) Improve the Methods section in particular on how the alpha-band analysis and how the permutation tests were formed.

2) Some of the claims must be adjusted accordingly to the outcome of the statistical test following the correction for multiple comparisons.

3) The claims on experiment 2 should be moderated in light of the limitations of the experiment (see points 1 and 2).

Reviewer #1:

I appreciate the authors' responsiveness to the first round of reviews, and it's clear they put a lot of hard work into running additional analyses and revising the manuscript.

While I think the manuscript is improved as a result, I remain unconvinced that there is unambiguous support for the conclusion that sequence context, rather than task demands more generally, reorganizes the structure of latent memory states. Additionally, a number of the new analyses included in the revision and some of the previously included analyses lack a comprehensive description in the method section, and these missing details make it impossible to evaluate some of the new results that were included in the revision. I've outlined these comments along with others in more detail below:

1. I think the additional analyses included in the revision convincingly rule out the alternative explanation that I raised in my initial review, which is that observers recoded the format of orientation stimuli in Experiment 2 and that such a recoding drives the difference in results across experiments.

We are glad that the reviewer is convinced that the new results ruled out the recoding interpretation raised in previous reviews.

2. However, I'm not convinced that these additional analyses rule out the point raised by Reviewer 3, which is that the comparison process that was necessary to complete the task in Experiment 2 was not necessary in Experiment 1. Therefore, it's possible that engaging in this comparison process (even if it doesn't reformat the representations) eliminates the backwards reactivations observed in Experiment 1 and that the backwards reactivation observed in Experiment 1 would occur any time two stimuli are observed in a sequence as long as they don't need to be directly compared. Thus, I'm not convinced that Experiment 2 supports the strong conclusion drawn in the manuscript that the need to store sequence information itself creates a unique latent state for sequence memory. However, I do agree with the authors that the reported work provides support for the more open-ended conclusion that different latent state representations are recruited based on task demands. I think revising the manuscript to emphasize this more modest claim throughout or providing a clear explanation that this limitation of Experiment 2 remains unresolved in the discussion would be an acceptable solution.

Thank you for raising the remaining concern. We agree with the reviewer that our current results so far could not completely exclude another interpretation, that is, the backward reactivation observed in Experiment 1 might be due to the lack of direct comparison task. In other words, Experiment 2 introduced additional task factor, i.e., comparison between memorized items, which might disrupt the backward reactivation observed in Experiment 1.

As suggested by the reviewer, we have added sentences explicitly discussing the limitation of Experiment 2 and future directions in Discussion (Page 16).

3. No details are provided in the method section of the revised manuscript about the alpha analyses that were included in the revision. How was the filtering done? Was the analysis run on total alpha-power as in most previous decoding work or power relative to some sort of baseline (i.e., percent change or DB2). What was the rationale behind using alpha-band power relative to baseline used to identify significant sensors before conducting the multivariate analysis?

We apologize for not including the methods about the alpha-band decoding analysis.

The time-frequency analysis was conducted using the continuous complex Gaussian wavelet transform (order = 4; for example, FWHM = 1.32 s for 1 Hz wavelet; Wavelet toolbox, MATLAB), with frequencies ranging from1 to 30 Hz, on each sensor, in each trial and in each subject separately, and the alpha-band (8 – 12 Hz) power time courses were then extracted from the output of the wavelet transform.

Yes, similar to most previous studies, the decoding analysis was performed on the total alpha-power rather than the relative power change.

About channel selection rationale, we are sorry for the confusion in previous version. In fact, we used the same posterior MEG channels selected in previous analysis to perform the alpha-band decoding analysis.

We have added more details and clarifications in Methods (Page 23 and 24). The figure legend has also been corrected (Figure 3-supplement 3).

4. How was the training and testing conducted for the newly added big and small label analyses? Were all trial types used to train the model and then big and small label orientations were tested separately? If not, using a common training set and then testing on each analysis set separately might improve the power of the analysis (see Sprague et al., 2018 for more details on the benefits of using a "fixed" encoding model). I might be missing something, but it seems like the orientation bin with the smallest angular distance from vertical would never have been included as a "big orientation" trial and the orientation bin with the largest angular distance from vertical would never be included for the "small orientation" decoding is that right? If so, please describe this in the manuscript, and along with how the IEM procedure was adapted to account for these missing orientations.

Thank you for raising the important question.

Yes, we used all the trial types to train the model and then tested big and small labels separately (i.e., fixed encoding model). Specifically, for the first orientation which was either labeled as ‘big’ or ‘small’ in each trial, we conducted the training on all trials and tested the big-labeled and small-labeled, separately. The same analysis was performed on the second orientation. Finally, the big/small orientation decoding results were combined.

The reviewer raised a critical concern that the big and small orientations might always belong to different trials. In fact, if we did training and testing on big- and small-labeled orientations separately, that would constitute an issue that might bias the decoding results. However, as stated above, we used all the trial types (e.g., the first orientation consists of big and small orientations with relative equal probability; same for the second orientation) to train the model and tested big and small labels separately and therefore would not be confounded by the possibility.

We thank the reviewer for the questions and have added more clarifications (Page 23).

5. The description of the cluster-based permutation test lines 633 – 639 is still difficult to follow and each step and decision needs to be laid out more clearly. It might help to cite a paper (or papers) that used the same approach applied here and to then report the thresholds set for the current manuscript. For example, some parts of the description sound like you used the approach applied by Sutterer et al. 2019 to identify clusters of above chance orientation selectivity and calculated a null distribution by first shuffling the orientation bin labels and re-running the IEM, repeating this process 1000 times and recording the largest cluster of above chance selectivity observed in each permutation. However, the current description of the analysis states that condition labels, rather than trial labels were shuffled. This approach would enable the identification of clusters where the selectivity differed between two conditions (e.g., big > small orientations), but it's not clear to me how this approach would allow for the identification of clusters of above chance selectivity within a single condition (i.e., when was orientation selectivity for big items higher than expected by chance). Finally, there should be some description of how the permutation test was adjusted for the addition and subtraction analyses. For instance, to calculate the null distribution for the additive analyses that were performed, permuted slope values for item 1 and item 2 (or the big and small orientations) should have been summed before calculating the permuted t-values, were they?

We apologize for the unclear explanations and have added a separate part with details and analysis steps in Methods (Page 22 and 23).

First, about the test statistics. As to the reactivation profiles, we first averaged the decoding performance over trials for each condition. Next, for each condition (first and second, or big and small), one-sample t-test for decoding performance against 0 was performed at each time point. As to the summed reactivation, the slope values of the two conditions were summed and the results were then compared against 0 using one-sample t-test, at each time point. As to the cross-condition reactivation comparison, paired t-test (second > first, or big > small within Experiment) or independent-sample t test (i.e., reactivation difference between Experiments) were performed on the reactivation profiles, at each time point.

Second, about the cluster-based permutation tests. We first identified clusters of contiguous significant time points (p < 0.05, two-tailed) from the calculated test statistics (specified above). Cluster-level statistics was then calculated by computing the size of cluster. Next, a Monte-Carlo randomization procedure was used (randomizing data across conditions for 1000 times) to estimate the significance probabilities for each cluster. For cross-condition comparison, condition labels were randomly shuffled between the two conditions. For single condition, 0 (slope value) with the same sample size was generated and shuffled with the original data. The cluster-level statistics was then calculated from the surrogate data to estimate significance probabilities for each original cluster.

Finally, we performed the same statistical procedure as Sutterer’s study (Plos Bio., 2019). As shown in Author response image 5, this analysis was actually more sensitive than ours (first item: -0.12 – -0.10 s, cluster p = 0.02, 0.05 – 0.39 s and 0.5 – 0.88 s and 1.13 – 1.4 s, cluster p <0.001; second item: 1.06 – 1.5 s and 1.57 – 1.86 s, cluster p < 0.001).

Author response image 5

Download asset Open asset

6. The manuscript and response letter refer to effects that were greater than.05 and/or that did not survive multiple comparisons corrections as significant. I know it's frustrating that some of these slopes look "close to significant" or "would have been significant without multiple comparisons correction", but there is no such thing as close to significant in frequentist statistics, so a cluster corrected p value of >.05 should not be reported as significant. The revised manuscript should be adjusted accordingly and arguments resting on the fact that non-significant time courses "look similar" should be removed. Below are a few instances of this that I noticed, but there may be more throughout the manuscript.

Thank you for the concern. According to the reviewer’s suggestions, we have revised all the statistical significance statement in a strict and consistent way, i.e., only stating cluster-corrected p valued of < 0.05 to be significant.

Moreover, we would like to emphasize that we have used a conservative criteria (two-tailed test with an alpha level of 0.05) while many studies used one-tailed test.

a. line 195 – 196 the reactivation of the first item was not significant following cluster correction. This isn't a huge problem since other analyses confirm the relationship between the reactivation of item one and behavioral performance, but it should be reported correctly.

Revised (Page 7).

b. Line 258, Decoding performance was not above chance after correcting for multiple comparisons for either item even after changing the permutation test from the two-tailed test used in Experiment 1 to a one-tailed test. Thus, it isn't informative to draw conclusions about whether the reactivation profile of the two items is similar when there is no evidence that each individual item was reactivated in the first place.

Revised (Page 9).

c. Line 266 only one of these windows is significant.

Revised (Page 10).

7. Line 369- 373: It would be nice to see more of the reasoning that the authors included in the response statement to explain to readers why the existence of this parallel active code does not pose a problem for the activity silent account.

Added (Page 13).

8. Line- 440 (data not shown): show the behavioral analysis as a supplemental figure or report the result in line if it's used to provide support an argument.

Added (Page 9).

References:

Sprague, T. C., Adam, K. C. S., Foster, J. J., Rahmati, M., Sutterer, D. W., and Vo, V. A. (2018). Inverted Encoding Models Assay Population-Level Stimulus Representations, Not Single-Unit Neural Tuning. Eneuro, 5(3), ENEURO.0098-18.2018.

Sutterer, D. W., Foster, J. J., Adam, K. C. S., Vogel, E. K., and Awh, E. (2019). Item-specific delay activity demonstrates concurrent storage of multiple active neural representations in working memory. PLoS Biology, 17(4), e3000239.

Author details

1. Qiaoli Huang

   1. School of Psychological and Cognitive Sciences, Peking University, Beijing, China
   2. PKU-IDG/McGovern Institute for Brain Research, Peking University, Beijing, China
   3. Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China

   Contribution

   Conceptualization, Formal analysis, Investigation, Visualization, Methodology, Writing - original draft

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4592-9270

2. Huihui Zhang

   1. School of Psychological and Cognitive Sciences, Peking University, Beijing, China
   2. PKU-IDG/McGovern Institute for Brain Research, Peking University, Beijing, China
   3. Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China

   Contribution

   Investigation, Visualization, Methodology, Writing - original draft

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-5420-4063

3. Huan Luo

   1. School of Psychological and Cognitive Sciences, Peking University, Beijing, China
   2. PKU-IDG/McGovern Institute for Brain Research, Peking University, Beijing, China
   3. Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China

   Contribution

   Conceptualization, Resources, Supervision, Funding acquisition, Investigation, Methodology, Writing - original draft, Project administration

   For correspondence

   huan.luo@pku.edu.cn

   Competing interests

   Reviewing editor, eLife

   "This ORCID iD identifies the author of this article:" 0000-0002-8349-9796

• 1,169

  Page views

• 225

  Downloads

• 2

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.