In natural environments, our senses are exposed to a colourful mix of sensory impressions. Behaviourally relevant stimuli can appear in varying contexts, such as variations in lighting, acoustics, stimulus position, or the presence of other stimuli. Different contexts may require different responses to the same stimulus, for example, when the behavioural task changes (context dependence). Alternatively, the same response may be required for different stimuli, for example, when the sensory context changes (context invariance). Recent advances have elucidated how context-dependent processing can be performed by recurrent feedback in neural circuits (Mante et al., 2013; Wang et al., 2018a; Dubreuil et al., 2020). In contrast, the role of feedback mechanisms in context-invariant processing is not well understood.

In the classical view, stimuli are hierarchically processed towards a behaviourally relevant percept that is invariant to contextual variations. This is achieved by extracting increasingly complex features in a feedforward network (Kriegeskorte, 2015; Zhuang et al., 2021; Yamins and DiCarlo, 2016). Models of such feedforward networks have been remarkably successful at learning complex perceptual tasks (LeCun et al., 2015), and they account for various features of cortical sensory representations (DiCarlo and Cox, 2007; Kriegeskorte et al., 2008; DiCarlo et al., 2012; Hong et al., 2016; Cichy et al., 2016). Yet, these models neglect feedback pathways, which are abundant in sensory cortex (Felleman and Van Essen, 1991; Markov et al., 2014) and shape sensory processing in critical ways (Gilbert and Li, 2013). Incorporating these feedback loops into models of sensory processing increases their flexibility and robustness (Spoerer et al., 2017; Alamia et al., 2021; Nayebi et al., 2021) and improves their fit to neural data (Kar et al., 2019; Kietzmann et al., 2019; Nayebi et al., 2021). At the neuronal level, feedback is thought to modulate rather than drive local responses (Sherman and Guillery, 1998), for instance, depending on behavioural context (Niell and Stryker, 2010; Vinck et al., 2015; Kuchibhotla et al., 2017; Dipoppa et al., 2018).

Here, we investigate the hypothesis that feedback modulation provides a neural mechanism for context-invariant perception. To this end, we trained a feedback-modulated network model to perform a context-invariant perceptual task and studied the resulting neural mechanisms. We show that the feedback modulation does not need to be temporally or spatially precise and can be realised by feedback-driven gain modulation in rate-based networks of excitatory and inhibitory neurons. To solve the task, the feedback loop dynamically maintains an invariant subspace in the population representation (Hong et al., 2016). This invariance is not present at the single-neuron level. Finally, we find that the feedback conveys a nonlinear representation of the context itself, which can be hard to discern by linear decoding methods.

These findings corroborate that feedback-driven gain modulation of feedforward networks enables context-invariant sensory processing. The underlying mechanism links single-neuron modulation with its function at the population level, highlighting the importance of population-level analyses.

As a simple instance of a context-invariant task, we considered a dynamic version of the blind source separation problem. The task is to recover unknown sensory sources, such as voices at a cocktail party (McDermott, 2009), from sensory stimuli that are an unknown mixture of the sources. In contrast to the classical blind source separation problem, the mixture can change in time, for example, when the speakers move around, thus providing a time-varying sensory context. Because the task requires a dynamic inference of the context, it cannot be solved by feedforward networks (Figure 1—figure supplement 1) or standard blind source separation algorithms (e.g. independent component analysis; Bell and Sejnowski, 1995; Hyvärinen and Oja, 2000). We hypothesised that this dynamic task can be solved by a feedforward network that is subject to modulation from a feedback signal. In our model, the feedback signal is provided by a modulatory system that receives both sensory stimuli and network output (Figure 1a).

Figure 1 with 6 supplements see all

Download asset Open asset

Invariance can be established by slow feedback modulation

Among the many modulatory mechanisms, even the faster ones are believed to operate on timescales of hundreds of milliseconds (Bang et al., 2020; Molyneaux and Hasselmo, 2002), raising the question if feedback-driven modulation is sufficiently fast to compensate for dynamic changes in environmental context.

To investigate how the timescale of modulation affects the performance in the dynamic blind source separation task, we trained network models, in which the modulatory feedback had an intrinsic timescale that forced it to be slow. We found that the signal clarity degraded only when this timescale was on the same order of magnitude as the timescale of contextual changes (Figure 2a). Note that timescales in this model are relative and could be arbitrarily rescaled. While slower feedback modulation produced a larger initial error (Figure 2b and c), it also reduced the fluctuations in the readout weights such that they more closely follow the optimal weights (Figure 2b). This speed-accuracy trade-off explains the lower and more variable signal clarity for slow modulation (Figure 2a) because the signal clarity was measured over the whole duration of a context and the transient onset error dominated over the reduced fluctuations.

Figure 2 with 1 supplement see all

Download asset Open asset

To determine architectural constraints on the modulatory system, we asked how these results depended on the input it received. So far, the modulatory system received the feedforward network’s inputs (the sensory stimuli) and its outputs (the inferred sources, see Figure 1a), but are both of these necessary to solve the task? We found that when the modulatory system only received the sensory stimuli, the model could still learn the task, though it was more sensitive to slow modulation (Figure 2d, Figure 2—figure supplement 1). When the modulatory system had to rely on the network output alone, task performance was impaired even for fast modulation (Figure 2e, Figure 2—figure supplement 1). Thus, while the modulatory system is more robust to slow modulation when it receives the network output, the output is not sufficient to solve the task.

Taken together, these results show that the biological timescale of modulatory mechanisms does not pose a problem for flexible feedback-driven processing as long as the feedback modulation changes on a faster timescale than variations in the context. In fact, slow modulation can increase processing accuracy by averaging out fluctuations in the feedback signal. Nevertheless, slow modulation likely requires the modulatory system to receive both input and output of the sensory system it modulates.

Invariance can be established by spatially diffuse feedback modulation

Neuromodulators are classically believed to diffusely affect large areas of the brain. Furthermore, signals in the brain are processed by populations of neurons. We wondered if the proposed modulation mechanism is consistent with such biological constraints. We therefore extended the network model such that the sensory stimuli are projected to a population of 100 neurons. A fixed linear readout of this population determined the network output. The neurons in the population received spatially diffuse modulatory feedback (Figure 3a) such that the feedback modulation affected neighbouring neurons similarly. We here assume that all synaptic weights to a neuron receive the same modulation, such that the feedback performs a gain modulation of neural activity (Ferguson and Cardin, 2020). The spatial specificity of the modulation was determined by the number of distinct feedback signals and their spatial spread (Figure 3b, Figure 3—figure supplement 1a).

Figure 3 with 1 supplement see all

Download asset Open asset

This population-based model with less specific feedback modulation could still solve the dynamic blind source separation task. The diffuse feedback modulation switched when the context changed, but was roughly constant within contexts (Figure 3c), as in the simple model. The effective weight from the stimuli to the network output also inverted the linear mixture of the sources (Figure 3—figure supplement 1d, compare with Figure 1c).

We found that only a few distinct feedback signals were needed for a clean separation of the sources across contexts (Figure 3d). Moreover, the feedback could have a spatially broad effect on the modulated population without degrading the signal clarity (Figure 3e, Figure 3—figure supplement 1), consistent with the low dimensionality of the context.

We conclude that, in our model, neuromodulation does not need to be spatially precise to enable flexible processing. Given that the suggested feedback-driven modulation mechanism works for slow and diffuse feedback signals, it could in principle be realised by neuromodulatory pathways present in the brain.

Invariance emerges at the population level

Having established that slow and spatially diffuse feedback modulation enables context-invariant processing, we next investigated the underlying mechanisms at the single-neuron and population level. Given that the readout of the population activity was fixed, it is not clear how the context-dependent modulation of single neurons could give rise to a context-independent network output. One possible explanation is that some of the neurons are context-invariant and are exploited by the readout. However, a first inspection of neural activity indicated that single neurons are strongly modulated by context (Figure 4a). To quantify this, we determined the signal clarity for each neuron at each stage of the feedforward network, averaged across contexts (Figure 4b). As expected, the signal clarity was low for the sensory stimuli. Intriguingly, the same was true for all neurons of the modulated neural population, indicating no clean separation of the sources at the level of single neurons. Although most neurons had a high signal clarity in some of the contexts, there was no group of neurons that consistently represented one or the other source (Figure 4c). Furthermore, the average signal clarity of the neurons did not correlate with their contribution to the readout (Figure 4d). Since single-neuron responses were not invariant, context invariance must arise at the population level.

Figure 4

Download asset Open asset

To confirm this, we asked how well the sources could be decoded at different stages of the feedforward network. We trained a single linear decoder of the sources on one set of contexts and tested its generalisation to novel contexts. We found that the decoding performance was poor for the sensory stimuli (Figure 4e), indicating that these did not contain a context-invariant representation. In contrast, the sources could be decoded with high accuracy from the modulated population.

This demonstrates that while individual neurons were not invariant, the population activity contained a context-invariant subspace. In fact, the population had to contain an invariant subspace because the fixed linear readout of the population was able to extract the sources across contexts. However, the linear decoding approach shows that this subspace can be revealed from the population activity itself with only a few contexts and no knowledge of how the neural representation is used downstream. The same approach could therefore be used to reveal context-invariant subspaces in neural data from population recordings. Note that the learned readout and the decoder obtained from population activity are not necessarily identical due to the high dimensionality of the population activity compared to the sources.

Feedback reorients the population representation

The question remains how exactly the context-invariant subspace is maintained by feedback modulation. In contrast to a pure feedforward model of invariant perception (Kriegeskorte, 2015; Yamins and DiCarlo, 2016), feedback-mediated invariance requires time to establish after contextual changes. Experimentally, hallmarks of this adaptive process should be visible when comparing the population representations immediately after a change and at a later point in time. Our model allows to cleanly separate the early and late representation by freezing the feedback signals in the initial period after a contextual change (Figure 5a), thereby disentangling the effects of feedback and context on population activity.

Figure 5 with 1 supplement see all

Download asset Open asset

The simulated experiment consisted of three stages: first, the feedback was intact for a particular context and the network outputs closely tracked the sources. Second, the context was changed but the feedback modulation was frozen at the same value as before. As expected, this produced deviations of the output from the sources. Third, for the same context the feedback modulation was turned back on, which reinstated the source signals in the output. In this experiment, we used pure sines as signals for visualisation purposes (Figure 5a and c). To visualise the population activity in the three stages of the experiment, we considered the space of the two readout dimensions and the first principal component (Figure 5b). We chose this space rather than, for example, the first three principal components (Figure 5—figure supplement 1), because it provides an intuitive illustration of the invariant subspace.

Because the sources were two-dimensional, the population activity followed a pattern within a two-dimensional subspace (Figure 5b, left; Figure 5—figure supplement 1a). For intact feedback, this population activity matched the sources when projected onto the readout (Figure 5c, left). Changing the context while freezing the feedback rotated and stretched this representation within the same subspace, such that the readout did not match the sources (Figure 5b and c, centre). Would turning the feedback modulation back on simply reverse this transformation to re-establish an invariant subspace? We found that this was not the case. Instead, the feedback rotated the representation out of the old subspace (Figure 5b, right), thereby reorienting it into the invariant readout (Figure 5c, right).

To quantify the transformation of the population representation, we repeated this experiment multiple times and determined the angle between the neural subspaces. Consistent with the illustration in Figure 5b, changing the context did not change the subspace orientation, whereas unfreezing the feedback caused a consistent reorientation (Figure 5d). The magnitude of this subspace reorientation depended on the similarity of the old and new context. Similar contexts generally evoked population activity with similar subspace orientations (Figure 5d and e). This highlights that there is a consistent mapping between contexts and the resulting low-dimensional population activity.

In summary, the role of feedback-driven modulation in our model is to reorient the population representation in response to changing contexts such that an invariant subspace is preserved.

The mechanism generalises to a hierarchical Dalean network

So far, we considered a linear network, in which neural activity could be positive and negative. Moreover, feedback modulation could switch the sign of the neurons’ downstream influence, which is inconsistent with Dale’s principle. We wondered if the same population-level mechanisms would operate in a Dalean network, in which feedback is implemented as a positive gain modulation. Although gain modulation is a broadly observed phenomenon that is attributed to a range of cellular mechanisms (Ferguson and Cardin, 2020; Salinas and Thier, 2000), its effect at the population level is less clear (Shine et al., 2021).

We extended the feedforward model as follows (Figure 6a): first, all neurons had positive firing rates. Second, we split the neural population ($z$ in the previous model) into a ‘lower-level’ ($z^{\mathrm{L}}$) and ‘higher-level’ population ($z^{\mathrm{H}}$). The lower-level population served as a neural representation of the sensory stimuli, whereas the higher-level population was modulated by feedback. This allowed a direct comparison between a modulated and an unmodulated neural population. It also allowed us to include Dalean weights between the two populations. Direct projections from the lower-level to the higher-level population were excitatory. In addition, a small population of local inhibitory neurons provided feedforward inhibition to the higher-level population. Third, the modulation of the higher-level population was implemented as a local gain modulation that scaled the neural responses. As a specific realisation of gain modulation, we assumed that feedback targeted inhibitory interneurons (e.g. in layer 1; Abs et al., 2018; Ferguson and Cardin, 2020; Cohen-Kashi Malina et al., 2021) that mediate the modulation in the higher-level population (e.g. via presynaptic inhibition; Pardi et al., 2020; Naumann and Sprekeler, 2020). This means that stronger feedback decreased the gain of neurons (Figure 4b). We will refer to these modulatory interneurons as modulation units $m$ (green units in Figure 4a).

Figure 6 with 1 supplement see all

Download asset Open asset

We found that this biologically more constrained model could still learn the context-invariant processing task (Figure 6—figure supplement 1a and b). Notably, the network’s performance did not depend on specifics of the model architecture, such as the target of the modulation or the number of inhibitory neurons (Figure 6—figure supplement 1c–e). In analogy to the previous model, the gain modulation of individual neurons changed with the context and thus enabled the flexible processing required to account for varying context (Figure 4c). The average gain over contexts was similar across neurons, whereas within a context the gains were broadly distributed (Figure 4d).

To verify if the task is solved by the same population-level mechanism, we repeated our previous analyses on the single-neuron and population level. Indeed, all results generalised to the Dalean network with feedback-driven gain modulation (compare with Figures 4—6). Single neurons in the higher- and lower-level population were not context-invariant (Figure 6e), but the higher-level population contained a context-invariant subspace (Figure 6f). This was not the case for the lower-level population, underscoring that invariant representations do not just arise from projecting the sensory stimuli into a higher dimensional space. Instead, the invariant subspace in the higher-level population was again maintained by the feedback modulation, which reoriented the population activity in response to context changes (Figure 6g).

Feedback conveys a nonlinear representation of the context

Since single neurons in the higher-level population were not invariant to context, the population representation must also contain contextual information. Indeed, contextual variables could be linearly decoded from the higher-level population activity (Figure 7a). In contrast, decoding the context from the lower-level population gave much lower accuracy. This shows that the contextual information is not just inherited from the sensory stimuli but conveyed by the feedback via the modulatory units. We therefore expected that the modulatory units themselves would contain a representation of the context. To our surprise, decoding accuracy on the modulatory units was low. This seems counterintuitive, especially since the modulatory units clearly covaried with the contextual variables (Figure 7b). To understand these seemingly conflicting results, we examined how the context was represented in the activity of the modulation units.

Figure 7

Download asset Open asset

We found that the modulation unit activity did encode the contextual variables, albeit in a nonlinear way (Figure 7c). The underlying reason is that the feedback modulation needs to remove contextual variations, which requires nonlinear computations. Specifically, the blind source separation task requires an inversion of the linear mixture of sources. Consistent with this idea, nonlinear decoding approaches performed better (Figure 7d). In fact, the modulatory units contained a linear representation of the ‘inverse context’ (i.e. the inverse mixing matrix, see ‘Materials and methods’).

In summary, the higher-level population provides a linear representation not only of the stimuli, but also of the context. In contrast, the modulatory units contained a nonlinear representation of the context, which could not be extracted by linear decoding approaches. We speculate that if contextual feedback modulation is mediated by interneurons in layer 1, they should represent the context in a nonlinear way.

Accumulating evidence suggests that sensory processing is strongly modulated by top-down feedback projections (Gilbert and Li, 2013; Keller and Mrsic-Flogel, 2018). Here, we demonstrate that feedback-driven gain modulation of a feedforward network could underlie stable perception in varying contexts. The feedback can be slow, spatially diffuse, and low-dimensional. To elucidate how the context invariance is achieved, we performed single-neuron and population analyses. We found that invariance was not evident at the single-neuron level, but only emerged in a subspace of the population representation. The feedback modulation dynamically transformed the manifold of neural activity patterns such that this subspace was maintained across contexts. Our results provide further support that gain modulation at the single-cell level enables nontrivial computations at the population level (Failor et al., 2021; Shine et al., 2021).

To study how feedback-driven modulation can enable flexible sensory processing, we built models of feedforward networks that are modulated by feedback. The feedback was dynamically generated by a modulatory system, which we implemented as a recurrent network. The weights of the recurrent network were trained such that the feedback modulation allowed the feedforward network to solve a flexible invariant processing task.

The dynamic blind source separation task

As an instance of flexible sensory processing, we used a dynamic variant of blind source separation. In classical blind source separation, two or more unknown time-varying sources $\overrightarrow{s}(t)$ need to be recovered from a set of observations (i.e. sensory stimuli) $\overrightarrow{x}(t)$. The sensory stimuli are composed of an unknown linear mixture of the sources such that $\overrightarrow{x}(t)=A\overrightarrow{s}(t)$ with a fixed mixing matrix $A$. Recovering the sources requires to find weights $W$ such that $W\overrightarrow{x}(t)\approx \overrightarrow{s}(t)$. Ideally, $W$ is equal to the pseudo-inverse of the unknown mixing matrix $A$, up to permutations.

In our dynamic blind source separation task, we model variations in the stimulus context by changing the linear mixture over time – albeit on a slower timescale than the time-varying signals. Thus, the sensory stimuli are constructed as

(1) $\overrightarrow{x}(t)=A(t)\overrightarrow{s}(t)+{\sigma }_n\overrightarrow{\xi }(t)\mathit{\hspace{1em}},$

where $A(t)$ is a time-dependent mixing matrix and ${\sigma }_n$ is the amplitude of additive white noise $\overrightarrow{\xi }(t)$. The time-dependent mixing matrix determines the current context and was varied in discrete time intervals ${\displaystyle n_t}$, meaning that the mixing matrix $A(t)$ (i.e. the context) was constant for ${\displaystyle n_t}$ samples before it changed. The goal of the dynamic blind source separation task is to recover the original signal sources $\overrightarrow{s}$ from the sensory stimuli $\overrightarrow{x}$ across varying contexts. Thus, the network model output needs to be invariant to the specific context of the sources. Note that while the context was varied, the sources themselves were the same throughout the task, unless stated otherwise. Furthermore, in the majority of experiments the number of source signals and sensory stimuli was $n_s=2$. A list of default parameters for the dynamic blind source separation task can be found in Table 1.

Table 1

Parameter	Symbol	Value
Number of signals	${\displaystyle n_s}$	2
Number of samples in context	${\displaystyle n_t}$	1000
Additive noise	${\displaystyle {\sigma }_n}$	0.001
Sampling frequency	${\displaystyle f_s}$	8 KHz

Source signals

Request a detailed protocol

As default source signals, we used two compositions of two sines each (‘chords’) with a sampling rate of $f_s=8000$ Hz that can be written as

(2) $s_1(t)=\sin \left(2\pi f_{11}t/f_s\right)+\sin \left(2\pi f_{12}t/f_s\right)$

(3) $s_2(t)=\sin \left(2\pi f_{21}t/f_s\right)+\sin \left(2\pi f_{22}t/f_s\right)$

with frequencies $f_{11}=100$ Hz, $f_{12}=125$ Hz, $f_{21}=150$ Hz, and $f_{22}=210$ Hz. Note that in our model we measure time as the number of samples from the source signals, meaning that timescales are relative and could be arbitrarily rescaled.

In Figure 5, we used pure sine signals with frequency $f$ for visualisation purposes: $s_i=\sin \left(2\pi ft/f_s\right)$. We also validated the model on signals that are not made of sine waves, as a sawtooth and a square wave signal (Figure 1—figure supplement 4). Unless stated otherwise, the same signals were used for training and testing the model.

Time-varying contexts

Request a detailed protocol

We generated the mixing matrix $A$ for each context by drawing random weights from a uniform distribution between 0 and 1, allowing only positive mixtures of the sources. Unless specified otherwise, we sampled new contexts for each training batch and for the test data, such that the training and test data followed the same distribution without necessarily being the same. The dimension of the mixing matrices was determined by number of signals ${\displaystyle n_s}$ such that $A$ was of shape $n_s\times n_s$. To keep the overall amplitude of the sensory stimuli in a similar range across different mixtures, we normalised the row sums of each mixing matrix to one. In the case of $n_s=2$, this implies that the contexts (i.e. the mixing matrices) are drawn from a two-dimensional manifold (see Figure 8, bottom left). In addition, we only used the randomly generated mixing matrices whose determinant was larger than some threshold value. We did this to ensure that each signal mixture was invertible and that the weights needed to invert the mixing matrix were not too extreme. A threshold value of 0.2 was chosen based on visual inspection of the weights from the inverted mixing matrix.

Figure 8

Download asset Open asset

The current manuscript is a computational study, so no data have been generated for this manuscript. Modelling code is available under https://github.com/sprekelerlab/feedback_modulation_Naumann21, (copy archived at swh:1:rev:05373b093803e464082ad5b9e8ab2dbbf43bb23e) upon publication.

1. 1. Abs E
   2. Poorthuis RB
   3. Apelblat D
   4. Muhammad K
   5. Pardi MB
   6. Enke L
   7. Kushinsky D
   8. Pu DL
   9. Eizinger MF
   10. Conzelmann KK
   11. Spiegel I
   12. Letzkus JJ

   (2018) Learning-Related Plasticity in Dendrite-Targeting Layer 1 Interneurons

   Neuron 100:684–699.

   https://doi.org/10.1016/j.neuron.2018.09.001

   • PubMed
   • Google Scholar
2. Preprint

   1. Alamia A
   2. Mozafari M
   3. Choksi B
   4. VanRullen R

   (2021) On the Role of Feedback in Visual Processing: A Predictive Coding Perspective

   arXiv.

   https://arxiv.org/abs/2106.04225

   • Google Scholar
3. 1. Azimi Z
   2. Barzan R
   3. Spoida K
   4. Surdin T
   5. Wollenweber P
   6. Mark MD
   7. Herlitze S
   8. Jancke D

   (2020) Separable gain control of ongoing and evoked activity in the visual cortex by serotonergic input

   eLife 9:e53552.

   https://doi.org/10.7554/eLife.53552

   • PubMed
   • Google Scholar
4. 1. Bang D
   2. Kishida KT
   3. Lohrenz T
   4. White JP
   5. Laxton AW
   6. Tatter SB
   7. Fleming SM
   8. Montague PR

   (2020) Sub-second Dopamine and Serotonin Signaling in Human Striatum during Perceptual Decision-Making

   Neuron 108:999–1010.

   https://doi.org/10.1016/j.neuron.2020.09.015

   • PubMed
   • Google Scholar
5. 1. Bayer HM
   2. Glimcher PW

   (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal

   Neuron 47:129–141.

   https://doi.org/10.1016/j.neuron.2005.05.020

   • PubMed
   • Google Scholar
6. 1. Bell AJ
   2. Sejnowski TJ

   (1995) An information-maximization approach to blind separation and blind deconvolution

   Neural Computation 7:1129–1159.

   https://doi.org/10.1162/neco.1995.7.6.1129

   • PubMed
   • Google Scholar
7. 1. Bengio Y
   2. Simard P
   3. Frasconi P

   (1994) Learning long-term dependencies with gradient descent is difficult

   IEEE Transactions on Neural Networks 5:157–166.

   https://doi.org/10.1109/72.279181

   • PubMed
   • Google Scholar
8. 1. Branco T
   2. Staras K

   (2009) The probability of neurotransmitter release: variability and feedback control at single synapses

   Nature Reviews. Neuroscience 10:373–383.

   https://doi.org/10.1038/nrn2634

   • PubMed
   • Google Scholar
9. 1. Brodbeck C
   2. Jiao A
   3. Hong LE
   4. Simon JZ

   (2020) Neural speech restoration at the cocktail party: Auditory cortex recovers masked speech of both attended and ignored speakers

   PLOS Biology 18:e3000883.

   https://doi.org/10.1371/journal.pbio.3000883

   • PubMed
   • Google Scholar
10. 1. Bronkhorst AW

    (2015) The cocktail-party problem revisited: early processing and selection of multi-talker speech

    Attention, Perception & Psychophysics 77:1465–1487.

    https://doi.org/10.3758/s13414-015-0882-9

    • PubMed
    • Google Scholar
11. 1. Carandini M
    2. Heeger DJ

    (2011) Normalization as a canonical neural computation

    Nature Reviews. Neuroscience 13:51–62.

    https://doi.org/10.1038/nrn3136

    • PubMed
    • Google Scholar
12. 1. Cherry EC

    (1953) Some Experiments on the Recognition of Speech, with One and with Two Ears

    The Journal of the Acoustical Society of America 25:975–979.

    https://doi.org/10.1121/1.1907229

    • Google Scholar
13. 1. Chittajallu R
    2. Pelkey KA
    3. McBain CJ

    (2013) Neurogliaform cells dynamically regulate somatosensory integration via synapse-specific modulation

    Nature Neuroscience 16:13–15.

    https://doi.org/10.1038/nn.3284

    • PubMed
    • Google Scholar
14. 1. Cichy RM
    2. Khosla A
    3. Pantazis D
    4. Torralba A
    5. Oliva A

    (2016) Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence

    Scientific Reports 6:1–13.

    https://doi.org/10.1038/srep27755

    • PubMed
    • Google Scholar
15. 1. Cohen-Kashi Malina K
    2. Tsivourakis E
    3. Kushinsky D
    4. Apelblat D
    5. Shtiglitz S
    6. Zohar E
    7. Sokoletsky M
    8. Tasaka G-I
    9. Mizrahi A
    10. Lampl I
    11. Spiegel I

    (2021) NDNF interneurons in layer 1 gain-modulate whole cortical columns according to an animal’s behavioral state

    Neuron 109:2150–2164.

    https://doi.org/10.1016/j.neuron.2021.05.001

    • PubMed
    • Google Scholar
16. 1. DiCarlo JJ
    2. Cox DD

    (2007) Untangling invariant object recognition

    Trends in Cognitive Sciences 11:333–341.

    https://doi.org/10.1016/j.tics.2007.06.010

    • PubMed
    • Google Scholar
17. 1. DiCarlo JJ
    2. Zoccolan D
    3. Rust NC

    (2012) How does the brain solve visual object recognition?

    Neuron 73:415–434.

    https://doi.org/10.1016/j.neuron.2012.01.010

    • PubMed
    • Google Scholar
18. 1. Dipoppa M
    2. Ranson A
    3. Krumin M
    4. Pachitariu M
    5. Carandini M
    6. Harris KD

    (2018) Vision and Locomotion Shape the Interactions between Neuron Types in Mouse Visual Cortex

    Neuron 98:602–615.

    https://doi.org/10.1016/j.neuron.2018.03.037

    • PubMed
    • Google Scholar
19. 1. Disney AA
    2. Aoki C
    3. Hawken MJ

    (2007) Gain modulation by nicotine in macaque v1

    Neuron 56:701–713.

    https://doi.org/10.1016/j.neuron.2007.09.034

    • PubMed
    • Google Scholar
20. 1. Douglas RJ
    2. Martin KAC

    (2004) Neuronal circuits of the neocortex

    Annual Review of Neuroscience 27:419–451.

    https://doi.org/10.1146/annurev.neuro.27.070203.144152

    • PubMed
    • Google Scholar
21. Preprint

    1. Dubreuil A
    2. Valente A
    3. Beiran M
    4. Mastrogiuseppe F
    5. Ostojic S

    (2020) Complementary Roles of Dimensionality and Population Structure in Neural Computations

    bioRxiv.

    https://doi.org/10.1101/2020.07.03.185942

    • Google Scholar
22. Preprint

    1. Failor SW
    2. Carandini M
    3. Harris KD

    (2021) Learning Orthogonalizes Visual Cortical Population Codes

    bioRxiv.

    https://doi.org/10.1101/2021.05.23.445338

    • Google Scholar
23. 1. Felleman DJ
    2. Van Essen DC

    (1991) Distributed hierarchical processing in the primate cerebral cortex

    Cerebral Cortex (New York, N.Y 1:1–47.

    https://doi.org/10.1093/cercor/1.1.1-a

    • PubMed
    • Google Scholar
24. 1. Ferguson KA
    2. Cardin JA

    (2020) Mechanisms underlying gain modulation in the cortex

    Nature Reviews. Neuroscience 21:80–92.

    https://doi.org/10.1038/s41583-019-0253-y

    • PubMed
    • Google Scholar
25. Conference

    1. Finn C
    2. Abbeel P
    3. Levine S

    (2017)

    Model-agnostic meta-learning for fast adaptation of deep networks

    In International Conference on Machine Learning. pp. 1126–1135.

    • Google Scholar
26. 1. Gers FA
    2. Schmidhuber J
    3. Cummins F

    (2000) Learning to forget: continual prediction with LSTM

    Neural Computation 12:2451–2471.

    https://doi.org/10.1162/089976600300015015

    • PubMed
    • Google Scholar
27. 1. Gilbert CD
    2. Li W

    (2013) Top-down influences on visual processing

    Nature Reviews. Neuroscience 14:350–363.

    https://doi.org/10.1038/nrn3476

    • PubMed
    • Google Scholar
28. 1. Glaser JI
    2. Benjamin AS
    3. Chowdhury RH
    4. Perich MG
    5. Miller LE
    6. Kording KP

    (2020) Machine Learning for Neural Decoding

    ENeuro 7:ENEURO.0506-19.2020.

    https://doi.org/10.1523/ENEURO.0506-19.2020

    • PubMed
    • Google Scholar
29. 1. Greff K
    2. Srivastava RK
    3. Koutnik J
    4. Steunebrink BR
    5. Schmidhuber J

    (2017) LSTM: A Search Space Odyssey

    IEEE Transactions on Neural Networks and Learning Systems 28:2222–2232.

    https://doi.org/10.1109/TNNLS.2016.2582924

    • PubMed
    • Google Scholar
30. 1. Halassa MM
    2. Sherman SM

    (2019) Thalamocortical Circuit Motifs: A General Framework

    Neuron 103:762–770.

    https://doi.org/10.1016/j.neuron.2019.06.005

    • PubMed
    • Google Scholar
31. 1. Har-Shai Yahav P
    2. Zion Golumbic E

    (2021) Linguistic processing of task-irrelevant speech at a cocktail party

    eLife 10:e65096.

    https://doi.org/10.7554/eLife.65096

    • PubMed
    • Google Scholar
32. 1. Hasselmo ME
    2. McGaughy J

    (2004) High acetylcholine levels set circuit dynamics for attention and encoding and low acetylcholine levels set dynamics for consolidation

    Progress in Brain Research 145:207–231.

    https://doi.org/10.1016/S0079-6123(03)45015-2

    • PubMed
    • Google Scholar
33. 1. Hochreiter S
    2. Schmidhuber J

    (1997) Long short-term memory

    Neural Computation 9:1735–1780.

    https://doi.org/10.1162/neco.1997.9.8.1735

    • PubMed
    • Google Scholar
34. Conference

    1. Hochreiter S
    2. Younger AS
    3. Conwell PR

    (2001) Learning to learn using gradient descent

    In International Conference on Artificial Neural Networks. pp. 87–94.

    https://doi.org/10.1007/3-540-44668-0

    • Google Scholar
35. 1. Hong H
    2. Yamins DLK
    3. Majaj NJ
    4. DiCarlo JJ

    (2016) Explicit information for category-orthogonal object properties increases along the ventral stream

    Nature Neuroscience 19:613–622.

    https://doi.org/10.1038/nn.4247

    • PubMed
    • Google Scholar
36. 1. Huber PJ

    (1964) Robust Estimation of a Location Parameter

    The Annals of Mathematical Statistics 35:73–101.

    https://doi.org/10.1214/aoms/1177703732

    • Google Scholar
37. 1. Hyvärinen A
    2. Oja E

    (2000) Independent component analysis: algorithms and applications

    Neural Networks 13:411–430.

    https://doi.org/10.1016/s0893-6080(00)00026-5

    • PubMed
    • Google Scholar
38. 1. Jarvis S
    2. Nikolic K
    3. Schultz SR

    (2018) Neuronal gain modulability is determined by dendritic morphology: A computational optogenetic study

    PLOS Computational Biology 14:e1006027.

    https://doi.org/10.1371/journal.pcbi.1006027

    • PubMed
    • Google Scholar
39. 1. Kar K
    2. Kubilius J
    3. Schmidt K
    4. Issa EB
    5. DiCarlo JJ

    (2019) Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior

    Nature Neuroscience 22:974–983.

    https://doi.org/10.1038/s41593-019-0392-5

    • PubMed
    • Google Scholar
40. 1. Kawai H
    2. Lazar R
    3. Metherate R

    (2007) Nicotinic control of axon excitability regulates thalamocortical transmission

    Nature Neuroscience 10:1168–1175.

    https://doi.org/10.1038/nn1956

    • PubMed
    • Google Scholar
41. 1. Keller GB
    2. Mrsic-Flogel TD

    (2018) Predictive Processing: A Canonical Cortical Computation

    Neuron 100:424–435.

    https://doi.org/10.1016/j.neuron.2018.10.003

    • PubMed
    • Google Scholar
42. 1. Kietzmann TC
    2. Spoerer CJ
    3. Sörensen LKA
    4. Cichy RM
    5. Hauk O
    6. Kriegeskorte N

    (2019) Recurrence is required to capture the representational dynamics of the human visual system

    PNAS 116:21854–21863.

    https://doi.org/10.1073/pnas.1905544116

    • PubMed
    • Google Scholar
43. Preprint

    1. Kingma DP
    2. Ba J

    (2014) Adam: A Method for Stochastic Optimization

    arXiv.

    https://arxiv.org/abs/1412.6980

    • Google Scholar
44. 1. Kriegeskorte N
    2. Mur M
    3. Bandettini PA

    (2008) Representational similarity analysis - connecting the branches of systems neuroscience

    Frontiers in Systems Neuroscience 2:4.

    https://doi.org/10.3389/neuro.06.004.2008

    • PubMed
    • Google Scholar
45. 1. Kriegeskorte N

    (2015) Deep Neural Networks: A New Framework for Modeling Biological Vision and Brain Information Processing

    Annual Review of Vision Science 1:417–446.

    https://doi.org/10.1146/annurev-vision-082114-035447

    • PubMed
    • Google Scholar
46. 1. Kuchibhotla KV
    2. Gill JV
    3. Lindsay GW
    4. Papadoyannis ES
    5. Field RE
    6. Sten TAH
    7. Miller KD
    8. Froemke RC

    (2017) Parallel processing by cortical inhibition enables context-dependent behavior

    Nature Neuroscience 20:62–71.

    https://doi.org/10.1038/nn.4436

    • PubMed
    • Google Scholar
47. 1. Larkum ME
    2. Senn W
    3. Lüscher HR

    (2004) Top-down dendritic input increases the gain of layer 5 pyramidal neurons

    Cerebral Cortex (New York, N.Y 14:1059–1070.

    https://doi.org/10.1093/cercor/bhh065

    • PubMed
    • Google Scholar
48. 1. Laviv T
    2. Riven I
    3. Dolev I
    4. Vertkin I
    5. Balana B
    6. Slesinger PA
    7. Slutsky I

    (2010) Basal GABA regulates GABA(B)R conformation and release probability at single hippocampal synapses

    Neuron 67:253–267.

    https://doi.org/10.1016/j.neuron.2010.06.022

    • PubMed
    • Google Scholar
49. 1. LeCun Y
    2. Bengio Y
    3. Hinton G

    (2015) Deep learning

    Nature 521:436–444.

    https://doi.org/10.1038/nature14539

    • PubMed
    • Google Scholar
50. 1. Lillicrap TP
    2. Santoro A
    3. Marris L
    4. Akerman CJ
    5. Hinton G

    (2020) Backpropagation and the brain

    Nature Reviews. Neuroscience 21:335–346.

    https://doi.org/10.1038/s41583-020-0277-3

    • PubMed
    • Google Scholar
51. Preprint

    1. Lohani S
    2. Moberly AH
    3. Benisty H
    4. Landa B
    5. Jing M
    6. Li Y
    7. Higley MJ
    8. Cardin JA

    (2020) Dual color mesoscopic imaging reveals spatiotemporally heterogeneous coordination of cholinergic and neocortical activity

    bioRxiv.

    https://doi.org/10.1101/2020.12.09.418632

    • Google Scholar
52. 1. Mante V
    2. Sussillo D
    3. Shenoy KV
    4. Newsome WT

    (2013) Context-dependent computation by recurrent dynamics in prefrontal cortex

    Nature 503:78–84.

    https://doi.org/10.1038/nature12742

    • PubMed
    • Google Scholar
53. 1. Markov NT
    2. Vezoli J
    3. Chameau P
    4. Falchier A
    5. Quilodran R
    6. Huissoud C
    7. Lamy C
    8. Misery P
    9. Giroud P
    10. Ullman S
    11. Barone P
    12. Dehay C
    13. Knoblauch K
    14. Kennedy H

    (2014) Anatomy of hierarchy: feedforward and feedback pathways in macaque visual cortex

    The Journal of Comparative Neurology 522:225–259.

    https://doi.org/10.1002/cne.23458

    • PubMed
    • Google Scholar
54. 1. Marques T
    2. Nguyen J
    3. Fioreze G
    4. Petreanu L

    (2018) The functional organization of cortical feedback inputs to primary visual cortex

    Nature Neuroscience 21:757–764.

    https://doi.org/10.1038/s41593-018-0135-z

    • PubMed
    • Google Scholar
55. 1. McAdams CJ
    2. Maunsell JH

    (1999) Effects of attention on orientation-tuning functions of single neurons in macaque cortical area V4

    The Journal of Neuroscience 19:431–441.

    https://doi.org/10.1523/JNEUROSCI.19-01-00431.1999

    • PubMed
    • Google Scholar
56. 1. McDermott JH

    (2009) The cocktail party problem

    Current Biology 19:R1024–R1027.

    https://doi.org/10.1016/j.cub.2009.09.005

    • PubMed
    • Google Scholar
57. 1. Miller RJ

    (1998) Presynaptic receptors

    Annual Review of Pharmacology and Toxicology 38:201–227.

    https://doi.org/10.1146/annurev.pharmtox.38.1.201

    • PubMed
    • Google Scholar
58. 1. Molyneaux BJ
    2. Hasselmo ME

    (2002) GABA(B) presynaptic inhibition has an in vivo time constant sufficiently rapid to allow modulation at theta frequency

    Journal of Neurophysiology 87:1196–1205.

    https://doi.org/10.1152/jn.00077.2001

    • PubMed
    • Google Scholar
59. 1. Naumann LB
    2. Sprekeler H

    (2020) Presynaptic inhibition rapidly stabilises recurrent excitation in the face of plasticity

    PLOS Computational Biology 16:e1008118.

    https://doi.org/10.1371/journal.pcbi.1008118

    • PubMed
    • Google Scholar
60. Software

    1. Naumann LB

    (2022) sprekelerlab/feedback_modulation_Naumann22, version swh:1:rev:05373b093803e464082ad5b9e8ab2dbbf43bb23e

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:ee3aa6ce292eea649252b7f5b3175b3561e7d8ed;origin=https://github.com/sprekelerlab/feedback_modulation_Naumann22;visit=swh:1:snp:7225dc0edc82b286290d94bb47dd51093317e3dc;anchor=swh:1:rev:05373b093803e464082ad5b9e8ab2dbbf43bb23e
61. Preprint

    1. Nayebi A
    2. Sagastuy-Brena J
    3. Bear DM
    4. Kar K
    5. Kubilius J
    6. Ganguli S
    7. Sussillo D
    8. DiCarlo JJ
    9. Yamins DL

    (2021) Goal-Driven Recurrent Neural Network Models of the Ventral Visual Stream

    bioRxiv.

    https://doi.org/10.1101/2021.02.17.431717

    • Google Scholar
62. 1. Niell CM
    2. Stryker MP

    (2010) Modulation of visual responses by behavioral state in mouse visual cortex

    Neuron 65:472–479.

    https://doi.org/10.1016/j.neuron.2010.01.033

    • PubMed
    • Google Scholar
63. 1. Oberfeld D
    2. Klöckner-Nowotny F

    (2016) Individual differences in selective attention predict speech identification at a cocktail party

    eLife 5:e16747.

    https://doi.org/10.7554/eLife.16747

    • PubMed
    • Google Scholar
64. 1. Olshausen BA
    2. Anderson CH
    3. Van Essen DC

    (1993) A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information

    The Journal of Neuroscience 13:4700–4719.

    https://doi.org/10.1523/JNEUROSCI.13-11-04700.1993

    • PubMed
    • Google Scholar
65. 1. Pardi MB
    2. Vogenstahl J
    3. Dalmay T
    4. Spanò T
    5. Pu DL
    6. Naumann LB
    7. Kretschmer F
    8. Sprekeler H
    9. Letzkus JJ

    (2020) A thalamocortical top-down circuit for associative memory

    Science (New York, N.Y.) 370:844–848.

    https://doi.org/10.1126/science.abc2399

    • PubMed
    • Google Scholar
66. 1. Parthasarathy A
    2. Hancock KE
    3. Bennett K
    4. DeGruttola V
    5. Polley DB

    (2020) Bottom-up and top-down neural signatures of disordered multi-talker speech perception in adults with normal hearing

    eLife 9:e51419.

    https://doi.org/10.7554/eLife.51419

    • PubMed
    • Google Scholar
67. Preprint

    1. Pascanu R
    2. Mikolov T
    3. Bengio Y

    (2013) On the Difficulty of Training Recurrent Neural Networks

    arXiv.

    https://arxiv.org/abs/1211.5063

    • Google Scholar
68. Conference

    1. Paszke A
    2. Gross S
    3. Massa F
    4. Lerer A
    5. Bradbury J
    6. Chanan G
    7. Killeen T
    8. Lin Z
    9. Gimelshein N
    10. Antiga L

    (2019)

    Pytorch: An imperative style, high-performance deep learning library

    Advances in Neural Information Processing Systems. pp. 8026–8037.

    • Google Scholar
69. 1. Pinto L
    2. Goard MJ
    3. Estandian D
    4. Xu M
    5. Kwan AC
    6. Lee SH
    7. Harrison TC
    8. Feng G
    9. Dan Y

    (2013) Fast modulation of visual perception by basal forebrain cholinergic neurons

    Nature Neuroscience 16:1857–1863.

    https://doi.org/10.1038/nn.3552

    • PubMed
    • Google Scholar
70. 1. Polack PO
    2. Friedman J
    3. Golshani P

    (2013) Cellular mechanisms of brain state-dependent gain modulation in visual cortex

    Nature Neuroscience 16:1331–1339.

    https://doi.org/10.1038/nn.3464

    • PubMed
    • Google Scholar
71. 1. Poorthuis RB
    2. Bloem B
    3. Schak B
    4. Wester J
    5. de Kock CPJ
    6. Mansvelder HD

    (2013) Layer-specific modulation of the prefrontal cortex by nicotinic acetylcholine receptors

    Cerebral Cortex (New York, N.Y) 23:148–161.

    https://doi.org/10.1093/cercor/bhr390

    • PubMed
    • Google Scholar
72. 1. Purushothaman G
    2. Marion R
    3. Li K
    4. Casagrande VA

    (2012) Gating and control of primary visual cortex by pulvinar

    Nature Neuroscience 15:905–912.

    https://doi.org/10.1038/nn.3106

    • PubMed
    • Google Scholar
73. 1. Quiroga RQ
    2. Reddy L
    3. Kreiman G
    4. Koch C
    5. Fried I

    (2005) Invariant visual representation by single neurons in the human brain

    Nature 435:1102–1107.

    https://doi.org/10.1038/nature03687

    • PubMed
    • Google Scholar
74. 1. Reynolds JH
    2. Heeger DJ

    (2009) The normalization model of attention

    Neuron 61:168–185.

    https://doi.org/10.1016/j.neuron.2009.01.002

    • PubMed
    • Google Scholar
75. 1. Riesenhuber M
    2. Poggio T

    (1999) Hierarchical models of object recognition in cortex

    Nature Neuroscience 2:1019–1025.

    https://doi.org/10.1038/14819

    • PubMed
    • Google Scholar
76. 1. Roth MM
    2. Dahmen JC
    3. Muir DR
    4. Imhof F
    5. Martini FJ
    6. Hofer SB

    (2016) Thalamic nuclei convey diverse contextual information to layer 1 of visual cortex

    Nature Neuroscience 19:299–307.

    https://doi.org/10.1038/nn.4197

    • PubMed
    • Google Scholar
77. 1. Sabatini BL
    2. Tian L

    (2020) Imaging Neurotransmitter and Neuromodulator Dynamics In Vivo with Genetically Encoded Indicators

    Neuron 108:17–32.

    https://doi.org/10.1016/j.neuron.2020.09.036

    • PubMed
    • Google Scholar
78. 1. Salinas E
    2. Abbott LF

    (1997) Invariant visual responses from attentional gain fields

    Journal of Neurophysiology 77:3267–3272.

    https://doi.org/10.1152/jn.1997.77.6.3267

    • PubMed
    • Google Scholar
79. 1. Salinas E
    2. Thier P

    (2000) Gain modulation: a major computational principle of the central nervous system

    Neuron 27:15–21.

    https://doi.org/10.1016/s0896-6273(00)00004-0

    • PubMed
    • Google Scholar
80. 1. Salinas E
    2. Sejnowski TJ

    (2001) Gain modulation in the central nervous system: where behavior, neurophysiology, and computation meet

    The Neuroscientist 7:430–440.

    https://doi.org/10.1177/107385840100700512

    • PubMed
    • Google Scholar
81. 1. Sampathkumar V
    2. Miller-Hansen A
    3. Sherman SM
    4. Kasthuri N

    (2021) Integration of signals from different cortical areas in higher order thalamic neurons

    PNAS 118:e2104137118.

    https://doi.org/10.1073/pnas.2104137118

    • PubMed
    • Google Scholar
82. 1. Sherman SM
    2. Guillery RW

    (1998) On the actions that one nerve cell can have on another: distinguishing “drivers” from “modulators.”

    PNAS 95:7121–7126.

    https://doi.org/10.1073/pnas.95.12.7121

    • PubMed
    • Google Scholar
83. 1. Sherman SM

    (2016) Thalamus plays a central role in ongoing cortical functioning

    Nature Neuroscience 19:533–541.

    https://doi.org/10.1038/nn.4269

    • PubMed
    • Google Scholar
84. 1. Shine JM
    2. Müller EJ
    3. Munn B
    4. Cabral J
    5. Moran RJ
    6. Breakspear M

    (2021) Computational models link cellular mechanisms of neuromodulation to large-scale neural dynamics

    Nature Neuroscience 24:765–776.

    https://doi.org/10.1038/s41593-021-00824-6

    • PubMed
    • Google Scholar
85. 1. Spoerer CJ
    2. McClure P
    3. Kriegeskorte N

    (2017) Recurrent Convolutional Neural Networks: A Better Model of Biological Object Recognition

    Frontiers in Psychology 8:1551.

    https://doi.org/10.3389/fpsyg.2017.01551

    • PubMed
    • Google Scholar
86. 1. Stroud JP
    2. Porter MA
    3. Hennequin G
    4. Vogels TP

    (2018) Motor primitives in space and time via targeted gain modulation in cortical networks

    Nature Neuroscience 21:1774–1783.

    https://doi.org/10.1038/s41593-018-0276-0

    • PubMed
    • Google Scholar
87. Preprint

    1. Thorat S
    2. Aldegheri G
    3. Kietzmann TC

    (2021) Category-Orthogonal Object Features Guide Information Processing in Recurrent Neural Networks Trained for Object Categorization

    arXiv.

    https://arxiv.org/abs/2111.07898

    • Google Scholar
88. 1. Thurley K
    2. Senn W
    3. Lüscher H-R

    (2008) Dopamine increases the gain of the input-output response of rat prefrontal pyramidal neurons

    Journal of Neurophysiology 99:2985–2997.

    https://doi.org/10.1152/jn.01098.2007

    • PubMed
    • Google Scholar
89. 1. Urban-Ciecko J
    2. Fanselow EE
    3. Barth AL

    (2015) Neocortical somatostatin neurons reversibly silence excitatory transmission via GABAb receptors

    Current Biology 25:722–731.

    https://doi.org/10.1016/j.cub.2015.01.035

    • PubMed
    • Google Scholar
90. 1. van den Brink RL
    2. Pfeffer T
    3. Donner TH

    (2019) Brainstem Modulation of Large-Scale Intrinsic Cortical Activity Correlations

    Frontiers in Human Neuroscience 13:340.

    https://doi.org/10.3389/fnhum.2019.00340

    • PubMed
    • Google Scholar
91. Book

    1. van Hemmen JL
    2. Sejnowski TJ

    (2006) How Does Our Visual System Achieve Shift and Size Invariance

    In: van Hemmen JL, editors. Problems in Systems Neuroscience. Oxford University Press. pp. 322–340.

    https://doi.org/10.1093/acprof:oso/9780195148220.003.0016

    • Google Scholar
92. 1. Vinck M
    2. Batista-Brito R
    3. Knoblich U
    4. Cardin JA

    (2015) Arousal and locomotion make distinct contributions to cortical activity patterns and visual encoding

    Neuron 86:740–754.

    https://doi.org/10.1016/j.neuron.2015.03.028

    • PubMed
    • Google Scholar
93. 1. Wang JX
    2. Kurth-Nelson Z
    3. Kumaran D
    4. Tirumala D
    5. Soyer H
    6. Leibo JZ
    7. Hassabis D
    8. Botvinick M

    (2018a) Prefrontal cortex as a meta-reinforcement learning system

    Nature Neuroscience 21:860–868.

    https://doi.org/10.1038/s41593-018-0147-8

    • PubMed
    • Google Scholar
94. 1. Wang J
    2. Narain D
    3. Hosseini EA
    4. Jazayeri M

    (2018b) Flexible timing by temporal scaling of cortical responses

    Nature Neuroscience 21:102–110.

    https://doi.org/10.1038/s41593-017-0028-6

    • PubMed
    • Google Scholar
95. 1. Werbos PJ

    (1990) Backpropagation through time: what it does and how to do it

    Proceedings of the IEEE 78:1550–1560.

    https://doi.org/10.1109/5.58337

    • Google Scholar
96. 1. Wiskott L
    2. Sejnowski TJ

    (2002) Slow feature analysis: unsupervised learning of invariances

    Neural Computation 14:715–770.

    https://doi.org/10.1162/089976602317318938

    • PubMed
    • Google Scholar
97. 1. Yamins DLK
    2. DiCarlo JJ

    (2016) Using goal-driven deep learning models to understand sensory cortex

    Nature Neuroscience 19:356–365.

    https://doi.org/10.1038/nn.4244

    • PubMed
    • Google Scholar
98. 1. Zhuang C
    2. Yan S
    3. Nayebi A
    4. Schrimpf M
    5. Frank MC
    6. DiCarlo JJ
    7. Yamins DLK

    (2021) Unsupervised neural network models of the ventral visual stream

    PNAS 118:e2014196118.

    https://doi.org/10.1073/pnas.2014196118

    • PubMed
    • Google Scholar

Decision letter after peer review:

[Editors’ note: the authors submitted for reconsideration following the decision after peer review. What follows is the decision letter after the first round of review.]

Thank you for submitting the paper "Invariant neural subspaces maintained by feedback modulation" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by a Senior Editor. The reviewers have opted to remain anonymous.

We are sorry to say that, after consultation with the reviewers, we have decided that this work will not be considered further for publication by eLife at this time.

While all three reviewers appreciated the novelty of the proposed computational role for feedback connections, they estimated that substantial additional work would be needed to establish more firmly the mechanisms underlying context–invariant processing and its biological relevance. Given the extent of the criticisms, we have decided to reject the paper. Should further analyses allow you to fully address these criticisms we would be open to a resubmission.

Reviewer #1:

One of the key questions in sensory neuroscience is how cortical networks extract invariant percepts from variable sensory inputs. Much of the existing literature focuses on the role of feed–forward hierarchical processing for extracting such invariances. The present study proposes an alternative mechanism based on top–down feedback. Focusing on the so–called source–separation, or cocktail–party problem, the manuscript shows how sources mixed in a context–dependent manner can be separated independently of context, using feed–forward networks modulated by top–down context–dependent inputs.

The manuscript starts with a simplified, abstract network, and then progressively moves to more biologically plausible ones. By performing population analyses of network activity, the authors then argue for a mechanism based on context–invariant subspaces.

Strengths of the paper:

– novel proposal for an important class of cortical computations

– very elegant formulation of the problem

– the writing style is very clear and appealing

– network implementations at different levels of biophysical realism.

Weaknesses of the paper:

– the announced mechanism, based on invariant subspaces, is not clearly explained and needs to be supported by additional evidence.

– how the network detects contextual changes does not seem to be explained

– the analyses of network activity, their rationale and the resulting conclusions are difficult to follow.

While I very much appreciated the novelty and the elegance of the approach developed in this paper, ultimately, I was left wondering how the networks perform their task.

– The title and abstract announce a mechanism based on invariant neural subspaces. Clearly, since the readout is fixed, there must be an invariant subspace, but the key question is how it is generated and maintained across contexts. In the Results, this mechanism is explained only briefly at the very end of the results, in connection to Figure 6, which seems to be just an illustration. The authors would need to unpack what precisely the mechanism is (not clear right now) and give more evidence for it.

– An important complementary issue is how the network detects context changes. The manuscript states that "feedback–mediated invariance requires time to establish after contextual changes" (lines 245–246), but how this works does not seem to be explained. What type of error signal does the network use to change the gains?

On a related note, is the network trained on all the contexts it sees during testing, or is it able to deal with totally novel contexts?

– The logic of the sequence of analysis (optogenetic manipulations; correlation; changes in gain…) is a bit difficult to follow and needs more motivation. In particular, why is the non–linear encoding of context important?

– It is a bit surprising that the analyses focus on the most complex version of the network to examine mechanisms. Presumably the simplified networks could be leveraged to identify and explain the mechanisms in a more transparent manner.

Reviewer #2:

The authors aim to explore an understudied potential function of feedback connections: providing context–independent sensory processing. Invariant sensory processing is frequently assumed to be carried out by feedforward processing and much of the study of feedback focuses on how feedback could implement context–dependent processing. This makes this study promising and relatively novel.

The strengths of this paper are that it demonstrates convincingly and using a variety of network architectures and feedback mechanisms that feedback modulations can indeed help a network read out sensory input in a context–independent way.

The weaknesses are in the analysis and comparisons of the various networks. While the basic finding that this invariance does not result from invariant activity on the individual neuron level is interesting and of value, the explanation that it instead leads to invariant population activity is almost tautological given the network architecture. It is also unclear how the simpler models the authors present are meant to provide insight on either the more biologically detailed hierarchical model or on real neural processing, especially given that the mode of modulation in the simplest model (re–weighting of feedforward weights) differs from that of the later models (re–weighting of neural activation). In this way I don't feel that the authors fully achieved their goal of describing the mechanism of feedback modulation.

The methods appear technically sound, but I am confused by some of the choices. For example, the authors start with a single layer network where feedback modulates the weights between the input and output. This is a different mechanism than the normal neuronal gain usually attributed to feedback. The authors then add more details to push the model more in the biological direction, but multiple details are sometimes added at once and the logic behind these choices isn't always clear. I believe the authors switch to using neuronal gain when they want to explore spatially correlated modulation, but they don't talk about neuronal modulation until they introduce their full hierarchical model. The hierarchical model also adds Dale's law and a separate inhibitory population but it is not clear why these details were added or if/how they change the function of the model in a way relevant to understanding feedback modulation. Even the use of a multi–layer model is not very well motivated given that they show that this task can be completed with a very small one layer model. The simplicity of the task has implications for understanding some of these findings as well. For example, to show that modulatory signals can be spatially correlated, the authors create a model with many more neurons than is needed to solve the task and show that the modulatory signal can target nearby cells in this population similarly without sacrificing performance. But the low dimensional nature of the modulatory signal is only really an issue of interest in the context of a higher dimensional task. As a thought experiment: if the 2 neurons in the original model were simply replicated to 50 each and each population of 50 neurons was given the same modulation, this would be essentially equivalent to the original 2 cell model, but under the logic of what the authors have shown here, would supposedly demonstrate that modulatory signals still work if low dimensional. In this way, that analysis fell short.

I think that this work may spur more interest in studying the role of feedback for invariant sensory processing, which would be a very productive outcome. Furthermore, the demonstration that the context signals cannot be linearly readout from the cells performing the modulation is an important lesson for the analysis of neural data. I also think further reflection on the finding that the modulatory network needs direct sensory input (more so even than the input from later processing stages) will be very important for understanding how this modulation works and how it relates to biological structures. As the authors note, this may mean that their model is more akin to inputs from higher order thalamic areas, though even that mapping is imperfect due to the lack of recurrence.

I think it would help the readability of the paper if the authors included a few more brief descriptions of the methods in the Results. For example, a better description of how the signals are generated, the fact that the networks are trained with a single set of signals only, etc. Also, there were points where it wasn't clear if a network was tested under different conditions or actually retrained for them (for example, in figure 2d/e). Also, the fact that the modulation went from being on the weights to on the neurons themselves was not made clear in section "Invariance can be established by spatially diffuse feedback modulation". I also found the schematic in Figure 1a a bit confusing. I don't know why x is represented as a question mark when it is a sum of the two signals. I'd prefer a diagram that makes the dimensionality of x clearer (relatedly, why are there only 3 weights from x to y when I believe it is a 2x2 matrix).

"While we trained the modulatory system using supervised learning, the contextual inference is performed by its dynamics without access to the target sources and thus unsupervised" I feel this could be read as saying that an actual unsupervised objective was used, when in fact only supervised learning took place, so I would suggest re–wording.

I didn't understand the claim about matched EI inputs and how it depends on using gain modulation. This should probably be expanded and related to the main questions of the paper or possibly removed.

Figure 4i seems to be the main demonstration that individual neural activity itself is not invariant to context. I'd like to see a more in–depth exploration of this. Particularly, if the readout only relied on a small handful of neurons then finding that the rest of the neurons are not context–invariant wouldn't prove that individual neural invariance is not a relevant mechanism. Given that the readout from this network is known, it would be particularly easy to determine if the heavily weighted neurons in particular are or are not context invariant.

In general, I don't understand why the authors use a separately trained linear readout when trying to show that the population activity at the final layer is invariant. They eventually acknowledge that "Since this readout is obtained from the data, this procedure does not require knowledge of the readout in the network model. Note that the trained decoder and the network readout are not necessarily identical" but they don't explain why they are using this alternative readout or what new insights its use adds. Particularly, the performance of the network indicates the there is some sort of context invariant read out possible from this population, yet the authors use this other readout in a way that is seemingly supposed to add something to the explanation.

Be sure to say what errorbars are based on in all figures.

"In our model, the mechanism needs to satisfy a few key

requirements: i) the modulation is not uniform across the population, ii) it operates on a timescale similar to that of changes in context, and iii) it is driven by feedback projections." I don't understand claim (iii). If anything, the results show the importance of the modulation being driven by feedforward sensory signals (figure 2d/e).

"In addition, feedback inputs from the sensory to the modulatory system allow a better control of the modulated network state." I don't see how the connections from a sensory system to a modulatory system are "feedback".

Reviewer #3:

I appreciate the didactic way in which the manuscript was written (and beautiful figures!), in particular the progression from a vanilla architecture towards the full fledged model with EI rectified neurons with spatially specific modulation. My main concerns (detailed below) are two–fold:

1. I felt that some extensions were not explicitly justified (e.g. why 2 layers instead of 2, etc)

2. I was expecting more 'reverse–engineering' of the mechanism through which the network accomplishes a context invariant projection. This is the main result of the paper, as reflected in the title, so I think it deserves more unpacking. Below I unpack these concerns, sometimes providing some suggestions to improve the motivation and clarity of the paper (without any particular order)

1. Overall, the architecture choices are a bit unjustified. In the extreme, wouldn't the LSTM alone solve the task? The addition of each feedforward layer should be better motivated (e.g. more biologically realistic? In what sense?). For example, why add an extra layer from extensions 2 and 3? If those are necessary, this should be explained. If they are not necessary, they should be removed.

2. 'Because the task requires a dynamic inference of the context, it cannot be solved by feedforward networks or standard blind source separation algorithms' I think the paper could be better motivated if this was shown explicitly with some examples.

3. A figure explicitly illustrating the training setup would help motivate what is trivially solved and what is actually challenging. For instance, in the main manuscript, it is not clear in which cases the network is trained and tested on the same contexts (ie A(t)) and which cases it is not. In the first case, the context can be easily inferred from x(t) but the latter is more challenging?

4. however I understand that the paper is already too long, Intra / extrapolation results deserve more spotlight and unpacking in my opinion. In general, if there is a lack of space, I would merge Figure 1 and figure 2 – and jump directly to extension 1 – and move most of figure 2 to sup.

5. Most important concern to me: Figure 6, in which the mechanism is revealed, deserves more quantifications to explicitly pinpoint the mechanism. Three suggestions come to mind:

a. Plot the 3 PCs components (instead of just 1) and show the readout in this space. The key result is that the readout is invariant to context and this is not clearly illustrated at the moment. Instead, what is shown is that the representation changes, but that it changes in a way that preserves invariance on the readout is not clearly highlighted.

b. The authors highlight that the network is not just reversing the new mixing coefficients and projecting the activity back into the 2d low manifold. Instead, it is rotating everything out of this manifold. My suggestion would be to show this alternatively explicitly. Is it actually possible? Relatedly, what happens if the context is changed back to context 1?

c. Finally, all the statements made about this figure should be quantified and not just illustrated for 1 trial.

[Editors’ note: The authors appealed the original decision. What follows is the authors’ response to the first round of review.]

Reviewer #1:

One of the key questions in sensory neuroscience is how cortical networks extract invariant percepts from variable sensory inputs. Much of the existing literature focuses on the role of feed–forward hierarchical processing for extracting such invariances. The present study proposes an alternative mechanism based on top–down feedback. Focusing on the so–called source–separation, or cocktail–party problem, the manuscript shows how sources mixed in a context–dependent manner can be separated independently of context, using feed–forward networks modulated by top–down context–dependent inputs.

The manuscript starts with a simplified, abstract network, and then progressively moves to more biologically plausible ones. By performing population analyses of network activity, the authors then argue for a mechanism based on context–invariant subspaces.

Strengths of the paper:

– novel proposal for an important class of cortical computations

– very elegant formulation of the problem

– the writing style is very clear and appealing

– network implementations at different levels of biophysical realism.

Weaknesses of the paper:

– the announced mechanism, based on invariant subspaces, is not clearly explained and needs to be supported by additional evidence.

– how the network detects contextual changes does not seem to be explained

– the analyses of network activity, their rationale and the resulting conclusions are difficult to follow.

We thank the reviewer for their interest in our work. We hope to have addressed all of the weaknesses listed above with the revision of the manuscript. In brief:

– We provide a more in-depth analysis of the mechanism based on invariant subspaces (see Change 2 above).

– We show and discuss what the network needs to detect context changes (see Change 4 above).

– We rearranged the results in the manuscript and added additional explanations to make the sequence of analyses and main findings clearer (see Change 1 and 3 above).

While I very much appreciated the novelty and the elegance of the approach developed in this paper, ultimately, I was left wondering how the networks perform their task.

– The title and abstract announce a mechanism based on invariant neural subspaces. Clearly, since the readout is fixed, there must be an invariant subspace, but the key question is how it is generated and maintained across contexts. In the Results, this mechanism is explained only briefly at the very end of the results, in connection to Figure 6, which seems to be just an illustration. The authors would need to unpack what precisely the mechanism is (not clear right now) and give more evidence for it.

We agree with the reviewer that the mechanism based on invariant subspaces needed more unpacking to underscore the main claims of the paper. As outlined in Change 2 above, we have extended our analyses of the mechanism and provided a quantification of the findings illustrated in the previous Figure 6.

– An important complementary issue is how the network detects context changes. The manuscript states that "feedback–mediated invariance requires time to establish after contextual changes" (lines 245–246), but how this works does not seem to be explained. What type of error signal does the network use to change the gains?

On a related note, is the network trained on all the contexts it sees during testing, or is it able to deal with totally novel contexts?

We thank the reviewer for pointing out that this was not sufficiently explained. As described in Change 4, we have added several sentences and a supplementary figure to clarify how the modulator maps its input to the correct feedback modulation:

“[…] The modulator therefore had to use its recurrent dynamics to determine the appropriate modulatory feedback for the time-varying context, based on the sensory stimuli and the network output. Put differently, the modulator had to learn an internal model of the sensory data and the contexts, and use it to establish the desired context invariance in the output.“ (ll. 57-61)

“Because the network had to learn an internal model of the task, we expected a limited degree of generalisation to new situations. Indeed, the network was able to interpolate between source frequencies seen during training (Supp. Figure S5), but failed on sources and contexts that were qualitatively different (Supp. Figure S6 b-d). The specific computations performed by the modulator are therefore idiosyncratic to the problem at hand. Hence, we did not investigate the internal dynamics of the modulator in detail, but concentrated on its effect on the feedforward network.” (ll. 85-91)

These excerpts should also answer the question regarding new contexts. Note that contexts were randomly sampled from a continuous context space for training and testing. The network is therefore not tested on the exact same contexts, but on contexts from the same distribution (unless specified otherwise). We have also clarified this in the methods section:

“Unless specified otherwise, we sampled new contexts for each training batch and for the test data, such that the training and test data followed the same distribution without necessarily being the same.” (ll. 494-497)

Regarding the question of the error signal: The modulator does not use an error signal, but computes a new mapping from its input to the correct modulation in response to context changes. Since the inputs are time-dependent the modulator needs to see a sufficient number of time-points before it can provide the appropriate feedback signal.

In additional analyses we found that a feedforward network can also solve the task, if it is given a time window of the sensory signals rather than a single moment in time. Such a network requires about the same number of timesteps from the stimuli (compare with Supp. Figure S1i and Supp. Figure S6a) as the modular-based architecture. This shows that the time it takes the network to infer the context from its input is not particular to our model but to the task, i.e. the statistics of the sources and contexts.

– The logic of the sequence of analysis (optogenetic manipulations; correlation; changes in gain…) is a bit difficult to follow and needs more motivation. In particular, why is the non–linear encoding of context important?

As described in the general answer, we agree that the sequence of analyses was not intuitive to the reader and thus rearranged the results in the new version of the manuscript. In particular, we have removed the optogenetic manipulations and considerations of E/I balance, since they are not central to the message of the paper (see Change 1 and 3 above).

Regarding the non-linear encoding of the context: We now include a deeper discussion of the results (see Change 3) that aim to clarify the relevance of these findings:

“In summary, the higher-population provides a linear representation not only of the stimuli, but also of the context. In contrast, the modulatory units contained a nonlinear representation of the context, which could not be extracted by linear decoding approaches. We speculate that if contextual feedback modulation is mediated by interneurons in layer 1, they should represent the context in a nonlinear way.” (ll. 288-292).

– It is a bit surprising that the analyses focus on the most complex version of the network to examine mechanisms. Presumably the simplified networks could be leveraged to identify and explain the mechanisms in a more transparent manner.

We want to thank the reviewer for this suggestion as we think it has significantly helped us to improve the structure of the manuscript. We now perform the single cell and population analyses on the network with spatially diffuse modulation (from Figure 3), as this is the simplest model comprising a neural population. We then verify that the findings hold for the Dalean network (new Figure 6). Details on the new order of the results can be found in the list of major changes above (specifically point 1).

Reviewer #2:

The authors aim to explore an understudied potential function of feedback connections: providing context–independent sensory processing. Invariant sensory processing is frequently assumed to be carried out by feedforward processing and much of the study of feedback focuses on how feedback could implement context–dependent processing. This makes this study promising and relatively novel.

The strengths of this paper are that it demonstrates convincingly and using a variety of network architectures and feedback mechanisms that feedback modulations can indeed help a network read out sensory input in a context–independent way.

The weaknesses are in the analysis and comparisons of the various networks. While the basic finding that this invariance does not result from invariant activity on the individual neuron level is interesting and of value, the explanation that it instead leads to invariant population activity is almost tautological given the network architecture. It is also unclear how the simpler models the authors present are meant to provide insight on either the more biologically detailed hierarchical model or on real neural processing, especially given that the mode of modulation in the simplest model (re–weighting of feedforward weights) differs from that of the later models (re–weighting of neural activation). In this way I don't feel that the authors fully achieved their goal of describing the mechanism of feedback modulation.

The methods appear technically sound, but I am confused by some of the choices. For example, the authors start with a single layer network where feedback modulates the weights between the input and output. This is a different mechanism than the normal neuronal gain usually attributed to feedback. The authors then add more details to push the model more in the biological direction, but multiple details are sometimes added at once and the logic behind these choices isn't always clear. I believe the authors switch to using neuronal gain when they want to explore spatially correlated modulation, but they don't talk about neuronal modulation until they introduce their full hierarchical model. The hierarchical model also adds Dale's law and a separate inhibitory population but it is not clear why these details were added or if/how they change the function of the model in a way relevant to understanding feedback modulation. Even the use of a multi–layer model is not very well motivated given that they show that this task can be completed with a very small one layer model. The simplicity of the task has implications for understanding some of these findings as well. For example, to show that modulatory signals can be spatially correlated, the authors create a model with many more neurons than is needed to solve the task and show that the modulatory signal can target nearby cells in this population similarly without sacrificing performance. But the low dimensional nature of the modulatory signal is only really an issue of interest in the context of a higher dimensional task. As a thought experiment: if the 2 neurons in the original model were simply replicated to 50 each and each population of 50 neurons was given the same modulation, this would be essentially equivalent to the original 2 cell model, but under the logic of what the authors have shown here, would supposedly demonstrate that modulatory signals still work if low dimensional. In this way, that analysis fell short.

I think that this work may spur more interest in studying the role of feedback for invariant sensory processing, which would be a very productive outcome. Furthermore, the demonstration that the context signals cannot be linearly readout from the cells performing the modulation is an important lesson for the analysis of neural data. I also think further reflection on the finding that the modulatory network needs direct sensory input (more so even than the input from later processing stages) will be very important for understanding how this modulation works and how it relates to biological structures. As the authors note, this may mean that their model is more akin to inputs from higher order thalamic areas, though even that mapping is imperfect due to the lack of recurrence.

We thank the reviewer for their thorough assessment of our work and for raising important concerns about the sequence of analyses and line of argumentation. In the following we first address what we think are the main concerns raised. After that, we respond to the specific recommendations in more detail.

Main concerns and answers:

1. Concern: The emergence of subspaces at the population level is not a surprising finding.

It is indeed not surprising that the neural population contains a context-invariant subspace. We do not think the existence of an invariant subspace is the main finding of our work, but rather how this subspace can be dynamically maintained by feedback modulation. Nevertheless, we think it is worth noting that the invariant subspace can not only be extracted with the readout learned during training, but also with a separate decoder that was trained on only a few contexts. To make clearer why this could be of interest, we have edited the respective text:

“In fact, the population had to contain an invariant subspace, because the fixed linear readout of the population was able to extract the sources across contexts. However, the linear decoding approach shows that this subspace can be revealed from the population activity itself with only a few contexts and no knowledge of how the neural representation is used downstream. The same approach could therefore be used to reveal context-invariant subspaces in neural data from population recordings.” (ll. 179-185)

2. Concern: It is unclear how simpler models give insight into the more complex models. Relatedly, some model choices are not well motivated.

We acknowledge that the connection between the models was not made sufficiently clear. To address this, we made several changes to the manuscript. First, we demonstrate explicitly that the network with spatial modulation finds an equivalent solution to the initial linear readout network. In particular, we show that the effective weights from the stimuli to the network output in the network with spatial modulation also follow the inverse of the mixing (Supp. Figure S8d) and describe this in the text:

“The diffuse feedback modulation switched when the context changed, but was roughly constant within contexts (Figure 3c), as in the simple model. The effective weight from the stimuli to the network output also inverted the linear mixture of the sources (Supp. Figure S8d, compare with Figure 1c).” (ll. 143-146)

Second, we perform the population level analyses on the simpler spatially modulated model and then verify that the same results hold for the Dalean network (see Major Change 1 and 3 above). This demonstrates that our findings are not a result of the specific architecture of the feedforward model. Third, we have added a few more sentences to motivate the distinction between the two neural populations in the Dalean model:

“We extended the feedforward model as follows (Figure 6a): First, all neurons had positive firing rates. Second, we split the neural population (z in the previous model) into a "lower-level" (z^L) and "higher-level" population (z^H). The lower-level population served as a neural representation of the sensory stimuli, whereas the higher-level population was modulated by feedback. This allowed a direct comparison between a modulated and an unmodulated neural population. It also allowed us to include Dalean weights between the two populations.” (ll. 236-242)

3. Concern: The feedback can only be low-dimensional, because the task is low-dimensional.

The reviewer is right that the low dimensionality of the task is why the feedback modulation can be low-dimensional. More precisely, it is the low dimensionality of the context that allows a low-dimensional feedback modulation. We acknowledge this in lines 148-150 of the manuscript:

“Moreover, the feedback could have a spatially broad effect on the modulated population without degrading the signal clarity (Figure 3e, Supp. Figure S6), consistent with the low dimensionality of the context.” (ll. 148-150)

The size of the neural populations that are necessary to solve the task are more related to the dimensionality of the stimuli or the degree of nonlinearity in the input-output mapping. Therefore, a low-dimensional and diffuse modulation may still be functional for more high-dimensional or nonlinear tasks, as long as the context remains low-dimensional. We also think that for higher dimensional inputs feedforward mechanisms could play a role in preprocessing them either directly towards invariance or into a form where the modulation can achieve it most effectively.

I think it would help the readability of the paper if the authors included a few more brief descriptions of the methods in the Results. For example, a better description of how the signals are generated, the fact that the networks are trained with a single set of signals only, etc. Also, there were points where it wasn't clear if a network was tested under different conditions or actually retrained for them (for example, in figure 2d/e). Also, the fact that the modulation went from being on the weights to on the neurons themselves was not made clear in section "Invariance can be established by spatially diffuse feedback modulation". I also found the schematic in Figure 1a a bit confusing. I don't know why x is represented as a question mark when it is a sum of the two signals. I'd prefer a diagram that makes the dimensionality of x clearer (relatedly, why are there only 3 weights from x to y when I believe it is a 2x2 matrix).

We thank the reviewer for providing specific recommendations on how to improve the readability of the manuscript. We have implemented this advice by adding the key equations of the setup to the results. We also added a methods figure that explicitly illustrates the task, the model and the training setup on a more technical level (see Figure 8). Furthermore, we adapted Figure 1a according to the reviewer’s recommendation.

Regarding the lack of clarity if the network was retrained or not, we hope that the additional analyses on the generalisation ability of the network help make this point clearer. To make explicit that networks were retrained for Figure 2 we modified the text to say:

“To investigate how the timescale of modulation affects the performance in the dynamic blind source separation task, we trained network models, in which the modulatory feedback had an intrinsic timescale that forced it to be slow.” (ll. 103-105)

Finally, it is correct that in Figure 3 the modulation went from modulating weights to modulating neurons. For the spatially diffuse modulation, we assumed that all weights to a neuron receive the same modulation. Since synaptic inputs are integrated linearly, this is equivalent to a neuronal gain modulation. We have added an explicit explanation to the results:

“We here assume that all synaptic weights to a neuron receive the same modulation, such that the feedback performs a gain modulation of neural activity (Ferguson and Cardin, 2020).” (ll. 137-139)

"While we trained the modulatory system using supervised learning, the contextual inference is performed by its dynamics without access to the target sources and thus unsupervised" I feel this could be read as saying that an actual unsupervised objective was used, when in fact only supervised learning took place, so I would suggest re–wording.

Good point. We have changed this sentence to make clear that the training of the network itself is unsupervised. It now reads:

“The modulator was trained using supervised learning. Afterwards, its weights were fixed and it no longer had access to the target sources (see Materials and methods, Figure 8). The modulator therefore had to use its recurrent dynamics to determine the appropriate modulatory feedback for the time-varying context, based on the sensory stimuli and the network output. Put differently, the modulator had to learn an internal model of the sensory data and the contexts, and use it to establish the desired context invariance in the output.” (ll. 55-61)

I didn't understand the claim about matched EI inputs and how it depends on using gain modulation. This should probably be expanded and related to the main questions of the paper or possibly removed.

Motivated by this recommendation and other comments we have removed these results from the paper in order to make more room for the central claims of the manuscript (see Change 3 above).

Figure 4i seems to be the main demonstration that individual neural activity itself is not invariant to context. I'd like to see a more in–depth exploration of this. Particularly, if the readout only relied on a small handful of neurons then finding that the rest of the neurons are not context–invariant wouldn't prove that individual neural invariance is not a relevant mechanism. Given that the readout from this network is known, it would be particularly easy to determine if the heavily weighted neurons in particular are or are not context invariant.

We thank the reviewer for this suggestion on how to further explore the lack of invariance of single neurons. These analyses are now performed on the simpler network in a new Figure 4 (see Change 1 above). We have extended our analyses and the text according to the reviewer’s suggestions:

“However, a first inspection of neural activity indicated that single neurons are strongly modulated by context (Figure 4a). To quantify this, we determined the signal clarity for each neuron at each stage of the feedforward network, averaged across contexts (Figure 4b). As expected, the signal clarity was low for the sensory stimuli. Intriguingly, the same was true for all neurons of the modulated neural population, indicating no clean separation of the sources at the level of single neurons. Although most neurons had a high signal clarity in some of the contexts, there was no group of neurons that consistently represented one or the other source (Figure 4c). Furthermore, the average signal clarity of the neurons did not correlate with their contribution to the readout (Figure 4d).” (ll. 161-170)

In general, I don't understand why the authors use a separately trained linear readout when trying to show that the population activity at the final layer is invariant. They eventually acknowledge that "Since this readout is obtained from the data, this procedure does not require knowledge of the readout in the network model. Note that the trained decoder and the network readout are not necessarily identical" but they don't explain why they are using this alternative readout or what new insights its use adds. Particularly, the performance of the network indicates the there is some sort of context invariant read out possible from this population, yet the authors use this other readout in a way that is seemingly supposed to add something to the explanation.

We agree with the reviewer that it was not sufficiently clear why we used the linear readout obtained from the data. The original idea was to highlight that these analyses can be done on neural data, because they do not require knowing the readout that is performed by downstream areas. We acknowledge that raising this point while explaining the feedback-freezing-experiment is confusing. To address this, we now use the model’s readout (former Figure 6, now Figure 5). In addition, we explicitly highlight that an invariant readout could be obtained from neural data (see response to public review above).

Be sure to say what errorbars are based on in all figures.

Thank you for pointing this out. We have added the respective information to the captions.

"In our model, the mechanism needs to satisfy a few key

requirements: i) the modulation is not uniform across the population, ii) it operates on a timescale similar to that of changes in context, and iii) it is driven by feedback projections." I don't understand claim (iii). If anything, the results show the importance of the modulation being driven by feedforward sensory signals (figure 2d/e).

Yes, that's a fair point. We have rephrased the respective sentence as follows:

“[…] iii) it is driven by a brain region that has access to the information needed to infer the context” (ll. 349-350)

"In addition, feedback inputs from the sensory to the modulatory system allow a better control of the modulated network state." I don't see how the connections from a sensory system to a modulatory system are "feedback".

We believe that our phrasing was unfortunate. We meant feedback from the feedforward network’s output. In the brain, this could correspond to higher-level sensory areas:

“In addition, feedback inputs from higher-level sensory areas to the modulatory system allow a better control of the modulated network state.” (ll. 375-377).

Reviewer #3:

I appreciate the didactic way in which the manuscript was written (and beautiful figures!), in particular the progression from a vanilla architecture towards the full fledged model with EI rectified neurons with spatially specific modulation. My main concerns (detailed below) are two–fold:

1. I felt that some extensions were not explicitly justified (e.g. why 2 layers instead of 2, etc)

2. I was expecting more 'reverse–engineering' of the mechanism through which the network accomplishes a context invariant projection. This is the main result of the paper, as reflected in the title, so I think it deserves more unpacking. Below I unpack these concerns, sometimes providing some suggestions to improve the motivation and clarity of the paper (without any particular order)

We thank the reviewer for their very constructive feedback and agree with both points made here. We hope to have addressed them with the revision of the manuscript, in particular Changes 1-3 (see above). Below we provide answers to the specific recommendations and questions in some more detail.

1. Overall, the architecture choices are a bit unjustified. In the extreme, wouldn't the LSTM alone solve the task? The addition of each feedforward layer should be better motivated (e.g. more biologically realistic? In what sense?). For example, why add an extra layer from extensions 2 and 3? If those are necessary, this should be explained. If they are not necessary, they should be removed.

We are glad that the reviewers have brought this to our attention. We have substantially reorganised the paper to clarify which architectural choices are relevant for which finding (Changes 1 and 3).

We modified the description of the Dalean network to make our model choices more transparent to the reader:

“We extended the feedforward model as follows (Figure 6a): First, all neurons had positive firing rates. Second, we split the neural population (z in the previous model) into a "lower-level" (z^L) and "higher-level" population (z^H). […] This allowed for a direct comparison between a modulated and an unmodulated neural population. It also allowed us to include Dalean weights between the two populations.” (ll. 236-242)

“[…] the higher-level population contained a context-invariant subspace (Figure 6f). This was not the case for the lower-level population, underscoring that invariant representations do not just arise from projecting the sensory stimuli into a higher dimensional space.” (ll. 262-266)

Furthermore, we decided to perform our single neuron and population analyses on the simpler network with spatially diffuse modulation and verify them on the Dalean network (see Change 1). We hope that the new manuscript structure and the additional analyses address the concerns raised above.

Regarding the question whether an LSTM alone can solve the task: yes it can. We tested this during the course of the project. This is not surprising, because an LSTM could in principle even learn the same architecture as we use here. Our architecture may even make it more difficult to solve the task since it contains more constraints. However, the focus of this project was not on whether a recurrent network can solve this task, but rather what role feedback modulation could play in (invariant) sensory processing. We have therefore decided not to include any results from a purely recurrent network.

2. 'Because the task requires a dynamic inference of the context, it cannot be solved by feedforward networks or standard blind source separation algorithms' I think the paper could be better motivated if this was shown explicitly with some examples.

We agree that this would help to motivate the architecture of our model and have included a new supplementary figure, in which we explicitly demonstrate that a feedforward network cannot solve the task (see Supp. Figure S1).

3. A figure explicitly illustrating the training setup would help motivate what is trivially solved and what is actually challenging. For instance, in the main manuscript, it is not clear in which cases the network is trained and tested on the same contexts (ie A(t)) and which cases it is not. In the first case, the context can be easily inferred from x(t) but the latter is more challenging?

We thought this was an excellent idea and have created a methods figure that illustrates the task, the model and the training setup explicitly (see Figure 8). Generally, we always sample new contexts when testing the model, from the same distribution as during training (unless stated otherwise). We now also show that the network does not generalise to out-of-distribution contexts or sensory stimuli (Supp. Figure S6, see also Change 4 above).

4. however I understand that the paper is already too long, Intra / extrapolation results deserve more spotlight and unpacking in my opinion. In general, if there is a lack of space, I would merge Figure 1 and figure 2 – and jump directly to extension 1 – and move most of figure 2 to sup.

We thank the reviewer for this suggestion. We removed some results and reorganised the remaining results such that there is more focus on the invariant subspaces. Hence, we do not feel that the manuscript has become substantially longer. Furthermore, we would prefer to only show the proof of concept in the first figure, in order to not overload the reader. If the reviewers find that the paper is much too long, we could move figure 2 to the supplementary material.

5. Most important concern to me: Figure 6, in which the mechanism is revealed, deserves more quantifications to explicitly pinpoint the mechanism. Three suggestions come to mind:

Thanks, these are great suggestions.

a. Plot the 3 PCs components (instead of just 1) and show the readout in this space. The key result is that the readout is invariant to context and this is not clearly illustrated at the moment. Instead, what is shown is that the representation changes, but that it changes in a way that preserves invariance on the readout is not clearly highlighted.

Unfortunately, using the space of the first 3 PCs instead of the first PC and the readout axes does not illustrate the invariant subspace in an intuitive way (see Change 2). Therefore, we decided to stick with the more unconventional space but verify our findings for PC space in a new supplementary figure (Supp. Figure S9). To better illustrate that the readout is context-invariant we plotted the projection of the subspace onto the readout into the 3D figure (Figure 5b) and make explicit that this is the same projection as shown in Figure 5c.

b. The authors highlight that the network is not just reversing the new mixing coefficients and projecting the activity back into the 2d low manifold. Instead, it is rotating everything out of this manifold. My suggestion would be to show this alternatively explicitly. Is it actually possible? Relatedly, what happens if the context is changed back to context 1?

c. Finally, all the statements made about this figure should be quantified and not just illustrated for 1 trial.

Regarding b and c: As described in Change 2, we have implemented the reviewer’s advice by adding a quantification to the former Figure 6 (now Figure 5) in terms of the angles between the subspaces (Figure 5d). We also show that the magnitude of this change depends on the similarity of the old and the new context (Figure 5dande), indicating that there is a consistent mapping between context and the low-dimensional population activity. Switching back to context 1 would therefore reinstate the original subspace, without hysteretic effects.

Author details

1. Laura B Naumann

   1. Modelling of Cognitive Processes, Technical University of Berlin, Berlin, Germany
   2. Bernstein Center for Computational Neuroscience, Berlin, Germany

   Contribution

   Conceptualization, Formal analysis, Investigation, Methodology, Project administration, Software, Visualization, Writing – original draft, Writing – review and editing

   For correspondence

   laura-bella.naumann@bccn-berlin.de

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-7919-7349

2. Joram Keijser

   Modelling of Cognitive Processes, Technical University of Berlin, Berlin, Germany

   Contribution

   Conceptualization, Methodology, Project administration, Supervision, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

3. Henning Sprekeler

   1. Modelling of Cognitive Processes, Technical University of Berlin, Berlin, Germany
   2. Bernstein Center for Computational Neuroscience, Berlin, Germany

   Contribution

   Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Writing – original draft, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-0690-3553

• 1,351

  Page views

• 266

  Downloads

• 1

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.