Working memory shapes neural geometry in human EEG over learning

  1. Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
  2. Department of Physiology, Anatomy and Genetics, University of Oxford, Oxford, United Kingdom
  3. Department of Engineering, University of Cambridge, Cambridge, United Kingdom
  4. School of Psychology, University of Nottingham, Nottingham, United Kingdom
  5. Department of Psychiatry, University of Oxford, Oxford, United Kingdom

Peer review process

Not revised: This Reviewed Preprint includes the authors’ original preprint (without revision), an eLife assessment, public reviews, and a provisional response from the authors.

Read more about eLife’s peer review process.

Editors

  • Reviewing Editor
    Roshan Cools
    Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands
  • Senior Editor
    Michael Frank
    Brown University, Providence, United States of America

Reviewer #1 (Public review):

Wojcik et al. conducted a working memory (WM) experiment in which participants had to press the right or left button after being presented with a square (upright) or diamond stimulus. The response mapping ('context') depended on a colour cue presented at the start of each trial. This results in an XOR task, requiring participants to integrate colour and shape information. Importantly, multiple colours could map onto the same context, allowing the authors to disentangle the (neural) representations of context from those of colour.

The authors report that participants learn the appropriate context mappings quickly over the course of the experiment. Neural context representation is evident in the WM delay and emerges later in the experiment, unlike colour representation, which is present only during colour presentation and does not evolve over experimental time. There are furthermore results on neural geometry (averaged cross-generalized decoding) and neural dimensionality (averaged decoding after shattering all task dimensions), which are somewhat harder to interpret.

Overall, the findings are likely Important, as they highlight the flexible and future-oriented nature of WM. The strength of support at the moment is incomplete: there are some loose ends on the context/colour generalization, and the evidence for the XOR neural representation is not (yet) well-established.

I have one (major) concern and several suggestions for improvement.

(1a) As the authors also acknowledge in several places, the XOR dimension is strongly correlated with motor responses, in any case toward the end of the task (and by definition for all correct trials). This should be dealt with properly. Right now, e.g. Figures 2g/i, 2h/j, 3e/g, 3f/h are highly similar, respectively, because of this strong collinearity. I would remove the semi-duplicate graphs and/or deal with this explicitly through some partial regression, trial selection, or similar (and report these correlations).

(1b) Most worrisome in this respect is that one of the key results presented is that XOR decoding increases with learning. But also task accuracy increases, meaning that the proportion of correct trials increases with learning, meaning that the XOR and motor regressors become more similar over experimental time. This means that any classifier picking up on motor signals will be better able to do so later on in the task than earlier on. (In other words, the XOR regressor may be a noisy version of the motor regressor early on, and a more precise version of the motor regressor later on.) Therefore, the increase in XOR decoding over experimental time may be (entirely) due to an increase in similarity between the XOR and motor dimensions. The authors should either rule out this explanation, and/or remove/tone down the conclusions regarding the XOR coding increase. (Note that the takeaway regarding colour/context generalization does not depend on this analysis, fortunately.) The absence of a change in motor decoding with learning (as reported on page 11) does not affect this potential confound; in fact it is made more likely with it.

(2) Bayes factors would be valuable in several places, especially with null results (p. 5) or cases with borderline-significant p-values.

(3) The authors' interpretation of the key results implies that the abstract coding learned over the task should be relevant for behaviour. The current results do not show a particularly strong behavioural relevance of coding, to put it mildly. It might be worth exploring whether neural coding expresses itself in reaction times, rather than (in)correct responses, and reflecting on the (lack of) behavioural relevance in the Discussion.

(4) All data and experiment/analysis code should be made available, in public repositories (i.e., not "upon request").

Reviewer #2 (Public review):

This manuscript describes an experiment in which subjects learned to apply an XOR rule in a task in which an initial color cue conditioned the instruction ("press left" or "press right") conveyed by a subsequent shape.

This manuscript gives the impression of being written to address a sophisticated computational framework, but the experiment was not designed to test this framework. Stated differently, the memory-as-resource-for-computations framework may not be needed to account for the results presented here. Variants of this task have been used for decades, often in the context of prospective processing, and although the authors emphasize a dimensionality reduction operation, the task may actually only require the recoding of retrospectively relevant sensory information into the prospectively relevant rule that is needed to guide the response on that trial. Consequently, many of the claims are only partially supported.

The framework invoked by the authors is summarized in the second paragraph of the manuscript:

"Insights from machine learning and computational neuroscience further highlight the idea that memory processes can be viewed as a resource for computations rather than a passive mechanism for storage (Dasgupta & Gershman, 2021; Ehrlich & Murray, 2022). In this light, working memory adapts computations to the current task demands (Dasgupta & Gershman, 2021); pre-computed information can be stored in working memory, and thus reduce the computation time at the moment of the decision (Braver, 2012; Hunt et al., 2021). This perspective is further supported by computational modelling of neural circuits that contends that working memory will change neural geometry in a way that supports the temporal decomposition of computations (Ehrlich & Murray, 2022). This work suggests that the computational load at the moment of action can be thus alleviated by decomposing complex operations into several simple problems solved sequentially in time."

However, the relevance, certainly the necessity, of this framework leads to mischaracterizations of some elements of the task (including about a hypothesis), the emphasis of constructs that don't actually exist in the task, some logical inconsistencies, and the repeated invocation of operations like "dimensionality reduction" despite the fact that the authors find no evidence for them.

Beginning with the final point, the task presented here is a variant of a Badre-style hierarchical control task, one requiring solution at the second order of abstraction (i.e., the color conditions the interpretation of the shape [2nd order], which then determines the correct response [1st order]. These operations can be accomplished without dimensionality reduction by simply carrying out the remapping instructed by each element. For example, on a trial beginning with a blue color cue, the subject can use a lookup table to translate this into the rule "square = left; diamond = right". When the shape is subsequently presented, the subject responds according to this rule. This is really no different from any of the several studies that have shown prospective recoding of information in working memory, including the work from the 1990s in nonhuman primates, and several subsequent studies using fMRI in humans beginning in the 2000s. Importantly, this account does not involve dimensionality reduction in any overt way. If it were the case that the more recent computational work indicates that this operation of "prospective recoding" does, in fact, entail dimensionality reduction on this type of task, that would be interesting. However, I don't see evidence that this is the case. Although the authors carry out several analyses of shattering dimensionality, I do not find any that track this measure across epochs within the trial, an approach that would presumably capture epoch-to-epoch dimensionality reduction, if it occurred.

With regard to mischaracterization of a hypothesis, the authors state: "We hypothesised that working memory processes control the dimensionality of neural representations by selecting features for maintenance. We tested this prediction by exploring the learning dynamics of the colour representation." However, what is described here is not a test of a prediction about dimensionality reduction. Rather, it's a test of a prediction that color decoding would not persist after color offset. To describe this as "dimensionality reduction" misrepresents/mischaracterizes what's happening, which is the translation of color (on any trial, a low-dimensional variable) into the rule that was cued by that color. It is a translation of what kind of information is being represented, as opposed to a dimensionality reduction applied to a representation.

With regard to constructs that don't actually exist, it is unclear what the reality is in the study of a "color pair"? I.e., because colors are never presented together, nor associated in some way, this would seem to be a device that's helpful to the authors for thinking about how their task might be solved, rather than a fundamental aspect of the task that the reader needs to understand. Furthermore, the example given here wasn't helpful for this reader. (What WAS helpful was the description of the two possible strategies and accompanying references to Mayr & Kleigel and to Vandierendonck.)

With regard to logical inconsistencies, one is the notion that color is irrelevant. This is not true, in a literal sense, because if every color cue were rendered as the same monochromatic patch, one wouldn't be able to solve the task. What the authors could do to make their point is perhaps refer to Strategy 1, which corresponds to a less efficient way to solve the task.

Also inconsistent is the relation of the present work to a previous study carried out by this group in nonhuman primates. That task did not include a working memory delay, and so this is difficult to reconcile the comparison that the authors draw with this task with the many suggestions that they make that it's something about WM, per se, that allows for the efficient performance of this task.

"Crucially, the irrelevant feature was only discarded during the delay after it entered working memory." This statement is in direct contradiction with the authors' own reporting of the results: "Decoding analyses demonstrated that colour information peaked in the early colour locked period of the trial and then rapidly declined over time to reach chance levels before the delay-locked period, 𝑐𝑙𝑢𝑠𝑡𝑒𝑟 1: 0.082 − 0.484 𝑚𝑠, 𝑝 = 0.006 (Fig. 2c)."

Other areas where I had difficulties include:

(1) "These results suggest that participants rapidly discarded irrelevant colour information. Only information relevant for performance (context) entered working memory and was maintained."
Although this may be the case, each of the four colors also instructed a rule, and so what's being documented in this study is the translation of a cue into a rule, not the transformation of a "meaningless color" into a "meaningful context." It is very possible that if the authors only used two colors, one for each rule (i.e., one for each "context"), they'd get the same decoding results.

(2) "A defining characteristic of low-dimensional task representations is that they can be easily cross-generalised to different sensory instances of the same task."
This result is difficult to reconcile with the loss of color decoding with color offset. Must it not mean that the rule is being represented differently when cued, e.g., by blue vs. by pink, or by green vs. by khaki? If this is true, then this would also argue against the idea of dimensionality reduction during the delay period, because subjects will, in effect, have swapped needing to represent one of four colors with needing to represent one of four rules.

(3) The authors assert that "cross-colour generalisation of context in the delay period is already implied by the significant context decoding combined with the absence of irrelevant colour coding."
This is contradicted, however, by the failure of the direct test of cross-color decoding!

(4) "Taken together, these findings imply that participants constructed abstract representations of task features but that the mechanism responsible for this transformation relied heavily on discarding colour information early in trial time."

This statement does not follow from the data because no mechanism is being directly measured. Rather, it's simply the case that after translating the color to a rule, the color is no longer needed and so is no longer kept in an active state. There is certainly no evidence for "heavy reliance".

Reviewer #3 (Public review):

Summary:

Wójcik and colleagues investigated how the maintenance of task information in working memory influences the dimensionality of task representations. The task required an exclusive-or (XOR) mapping as the output by combining stimulus features separated by a delay period. The authors found that context information invariant to input features (i.e., color) is maintained and enhanced over the course of learning the task.

The significance of this study lies in its demonstration of how learning selectively changes the geometry of task representations. The clear-cut results emphasize that learning promotes the abstraction of task representations for context-dependent computations. It is also important to investigate how working memory mechanisms contribute to the geometry and optimization of task representations, as such studies in humans are scarce.

Strengths:

(1) The task design and analyses are clear.

(2) The theoretical motivation to study low-dimensional representations and temporal decomposition is strong. Understanding how learning changes these qualities is a novel and important question.

Weaknesses:

(1) The specific contribution of working memory maintenance to the dimensionality and abstraction of representations is unclear. While the task likely recruits working memory, there are no direct assessments linking the observed results to particular qualities or mechanisms of working memory. In other words, neural representations observed during the delay period are interpreted as working memory.

(2) The dissociation between XOR and motor representations is ambiguous, as they only become distinguishable during error trials. Additionally, they show similar time courses and learning-related changes.

Author Response:

Reviewer #1( Public review):

The reviewer raised two main concerns: the potential confound between XOR and motor coding, and the relationship between neural coding and behaviour.

First, we appreciate the consideration of the collinearity between the XOR and motor dimensions. We fully agree that this confound may have contributed to the observed increase in XOR decoding over the course of learning. In response, we will merge the XOR and motor features in the main figures, tone down our interpretation of the XOR learning effect, and clarify how motor signals may obscure or mimic XOR-related changes. As the reviewer noted, this confound does not affect the colour/context cross-generalisation analyses, which remain central to our conclusions regarding flexible and prospective working memory coding.

We also thank the reviewer for the suggestion to examine the behavioural relevance of the neural representations more directly. We agree entirely, and will incorporate new analyses relating coding strength to reaction times, as well as reflect on the implications of these results in the revised Discussion.

Reviewer #2 (Public Review):

The reviewer rightly noted that our manuscript overlooks the established concept of retrospective/prospective coding in working memory, giving the impression that we attempted to reframe it using newer machine learning terminology. We thank the reviewer for catching this important omission. Our intention was not to override this well-established conceptual framework with a newer machine learning term, but rather to build upon it. In fact, prospective coding and the idea of working memory as a resource for computation are closely related—one helps define the functions (prospective and retrospective coding) and the other explains the computational rationale behind applying them. For example, prospective codes specify what is being stored (future-relevant information), while the “memory-as-computation” view addresses why such representation is useful: to enable temporal decomposition of complex tasks and reduce computational load at decision time. We will revise the relevant paragraphs to explicitly reference this cognitive framework and clarify how it relates to — and is complemented by — the newer computational perspective we introduce. Thank you again for highlighting this.

Reviewer 2 also argues that the evidence presented does not support dimensionality reduction, noting that participants likely transition from processing the sensory cue (e.g., blue) to a rule-based representation (e.g., context 1 vs context 2) later in the trial, and that this remapping does not inherently require dimensionality reduction. We agree that our results are consistent with such a transformation into an abstract rule representation during the delay period, as supported by the observed cross- colour context generalisation (Figure 3b) and that this process does not require dimensionality reduction per se. However, we would like to clarify that a shared decision boundary between two colour pairs (e.g., context 1 vs context 2) can manifest in two types of neural geometries. In one case — observed in our data — the irrelevant colour dimension is not maintained after the presentation period, such that blue and pink are maintained as context 1 but variance along the blues vs pink dimension is not represented in neural activity. In the other case, it is possible for the same abstract rule (context 1) to be constructed while maintaining the sensory representation of colour (e.g., “blue” or “pink”), resulting in a change in representational geometry without a reduction in dimensionality. Our data do not support the latter scenario: irrelevant colour information is not maintained in the delay period, suggesting that the abstraction is accompanied by a loss of variance along irrelevant sensory dimensions—i.e., a form of dimensionality reduction. We will clarify this point in the revised manuscript and include a new analysis that explicitly tests whether shattering dimensionality changes as a function of trial time.

The reviewer also raised concerns about inconsistencies in our terminology, particularly the use of “colour pair” and “irrelevant colour.” We agree with the reviewer that the term “colour pair” was a conceptual device rather than a literal aspect of the task, and we will revise the text to make this clear. We recognise that our wording around “irrelevant colour” might have caused confusion. We did not mean “colour” in the broad sense of all colour processing, but rather referred to specific colour dimensions that are not relevant for task performance—for example, when context 1 is cued by both pink and blue, the dimension carrying variance between blue and pink can be considered irrelevant. We will clarify this point in the revised manuscript, using the reviewer’s suggestion to incorporate the description we had already provided in the Methods section.

While we respectfully disagree with the reviewer’s interpretation of our findings—particularly regarding the absence of dimensionality reduction, which they associate with the failure of the direct test of cross-colour context decoding (see Fig. 3b, which shows a significant effect)—we appreciate the opportunity to clarify our position and will revise the manuscript to ensure our reasoning is as transparent and rigorous as possible.

Reviewer #3 (PublIc review):

The reviewer values the study’s demonstration that learning promotes abstraction in task representations, but raises concerns about the lack of direct evidence linking delay-period activity to specific working memory mechanisms and the ambiguous dissociation between XOR and motor representations. We thank the reviewer for their careful reading of the manuscript and will address both concerns in the revised version. As mentioned in our response to Reviewer #1, we will merge the motor and XOR analyses, tone down our interpretations, and clarify why these signals are entangled. Additionally, we will link delay-period neural activity to behavioural performance to establish a more direct connection to working memory processes. Notably, in Figure 4f, we show that early in learning, participants who exhibit stronger cross-generalisation of context during the delay are also more likely to exhibit decreased shattering dimensionality at decision time — providing an early link between the preparation of a contextual signal and the subsequent reduction in computational complexity at decision time. We will include additional analyses to further strengthen this link in the revised manuscript.

  1. Howard Hughes Medical Institute
  2. Wellcome Trust
  3. Max-Planck-Gesellschaft
  4. Knut and Alice Wallenberg Foundation