Nonlinear transient amplification in recurrent neural networks with short-term plasticity

Version of Record

Accepted for publication after peer review and revision.

Download
Cite
Share
CommentOpen annotations (there are currently 0 annotations on this page).

Version of Record published: February 7, 2022 (This version)
Accepted Manuscript published: December 13, 2021 (Go to version)
Accepted: December 10, 2021
Received: June 14, 2021
Preprint posted: June 10, 2021 (Go to version)

1. Of interest
Dopamine in the dorsal bed nucleus of stria terminalis signals Pavlovian sign-tracking and reward violations

Utsav Gyawali, David A Martin ... Donna Calu

Research Article May 26, 2023
Further reading

Article
Figures and data
Abstract
Editor's evaluation
Introduction
Results
Discussion
Materials and methods
Appendix 1
Appendix 2
Appendix 3
Appendix 4
Appendix 5
Data availability
References
Decision letter
Author response
Article and author information
Metrics

Abstract

To rapidly process information, neural circuits have to amplify specific activity patterns transiently. How the brain performs this nonlinear operation remains elusive. Hebbian assemblies are one possibility whereby strong recurrent excitatory connections boost neuronal activity. However, such Hebbian amplification is often associated with dynamical slowing of network dynamics, non-transient attractor states, and pathological run-away activity. Feedback inhibition can alleviate these effects but typically linearizes responses and reduces amplification gain. Here, we study nonlinear transient amplification (NTA), a plausible alternative mechanism that reconciles strong recurrent excitation with rapid amplification while avoiding the above issues. NTA has two distinct temporal phases. Initially, positive feedback excitation selectively amplifies inputs that exceed a critical threshold. Subsequently, short-term plasticity quenches the run-away dynamics into an inhibition-stabilized network state. By characterizing NTA in supralinear network models, we establish that the resulting onset transients are stimulus selective and well-suited for speedy information processing. Further, we find that excitatory-inhibitory co-tuning widens the parameter regime in which NTA is possible in the absence of persistent activity. In summary, NTA provides a parsimonious explanation for how excitatory-inhibitory co-tuning and short-term plasticity collaborate in recurrent networks to achieve transient amplification.

Editor's evaluation

Many brain circuits, particularly those found in mammalian sensory cortices, need to respond rapidly to stimuli while at the same time avoiding pathological, runaway excitation. Over several years, many theoretical studies have attempted to explain how cortical circuits achieve these goals through interactions between inhibitory and excitatory cells. This study adds to this literature by showing how synaptic short-term depression can stabilise strong positive feedback in a circuit under a variety of plausible scenarios, allowing strong, rapid and stimulus-specific responses.

https://doi.org/10.7554/eLife.71263.sa0

Introduction

Perception in the brain is reliable and strikingly fast. Recognizing a familiar face or locating an animal in a picture only takes a split second (Thorpe et al., 1996). This pace of processing is truly remarkable since it involves several recurrently connected brain areas each of which has to selectively amplify or suppress specific signals before propagating them further. This processing is mediated through circuits with several intriguing properties. First, excitatory-inhibitory (EI) currents into individual neurons are commonly correlated in time and co-tuned in stimulus space (Wehr and Zador, 2003; Froemke et al., 2007; Okun and Lampl, 2008; Hennequin et al., 2017; Rupprecht and Friedrich, 2018; Znamenskiy et al., 2018). Second, neural responses to stimulation are shaped through diverse forms of short-term plasticity (STP) (Tsodyks and Markram, 1997; Markram et al., 1998; Zucker and Regehr, 2002; Pala and Petersen, 2015). Finally, mounting evidence suggests that amplification rests on neuronal ensembles with strong recurrent excitation (Marshel et al., 2019; Peron et al., 2020), whereby excitatory neurons with similar tuning preferentially form reciprocal connections (Ko et al., 2011; Cossell et al., 2015). Such predominantly symmetric connectivity between excitatory cells is consistent with the notion of Hebbian cell assemblies (Hebb, 1949), which are considered an essential component of neural circuits and the putative basis of associative memory (Harris, 2005; Josselyn and Tonegawa, 2020). Computationally, Hebbian cell assemblies can amplify specific activity patterns through positive feedback, also referred to as Hebbian amplification. Based on these principles, several studies have shown that Hebbian amplification can drive persistent activity that outlasts a preceding stimulus (Hopfield, 1982; Amit and Brunel, 1997; Yakovlev et al., 1998; Wong and Wang, 2006; Zenke et al., 2015; Gillary et al., 2017), comparable to selective delay activity observed in the prefrontal cortex when animals are engaged in working memory tasks (Funahashi et al., 1989; Romo et al., 1999).

However, in most brain areas, evoked responses are transient and sensory neurons typically exhibit pronounced stimulus onset responses, after which the circuit dynamics settle into a low-activity steady-state even when the stimulus is still present (DeWeese et al., 2003; Mazor and Laurent, 2005; Bolding and Franks, 2018). Preventing run-away excitation and multi-stable attractor dynamics in recurrent networks requires powerful and often finely tuned feedback inhibition resulting in EI balance (Amit and Brunel, 1997; Compte et al., 2000; Litwin-Kumar and Doiron, 2012; Ponce-Alvarez et al., 2013; Mazzucato et al., 2019), However, strong feedback inhibition tends to linearize steady-state activity (van Vreeswijk and Sompolinsky, 1996; Baker et al., 2020). Murphy and Miller, 2009 proposed balanced amplification which reconciles transient amplification with strong recurrent excitation by tightly balancing recurrent excitation with strong feedback inhibition (Goldman, 2009; Hennequin et al., 2012; Hennequin et al., 2014; Bondanelli and Ostojic, 2020; Gillett et al., 2020). Importantly, balanced amplification was formulated for linear network models of excitatory and inhibitory neurons. Due to linearity, it intrinsically lacks the ability to nonlinearly amplify stimuli which limits its capabilities for pattern completion and pattern separation. Further, how balanced amplification relates to nonlinear neuronal activation functions and nonlinear synaptic transmission as, for instance, mediated by STP (Tsodyks and Markram, 1997; Markram et al., 1998; Zucker and Regehr, 2002; Pala and Petersen, 2015), remains elusive. This begs the question of whether there are alternative nonlinear amplification mechanisms and how they relate to existing theories of recurrent neural network processing.

Here, we address this question by studying an alternative mechanism for the emergence of transient dynamics that relies on recurrent excitation, supralinear neuronal activation functions, and STP. Specifically, we build on the notion of ensemble synchronization in recurrent networks with STP (Loebel and Tsodyks, 2002; Loebel et al., 2007) and study this phenomenon in analytically tractable network models with rectified quadratic activation functions (Ahmadian et al., 2013; Rubin et al., 2015; Hennequin et al., 2018; Kraynyukova and Tchumatchenko, 2018) and STP. We first characterize the conditions under which individual neuronal ensembles with supralinear activation functions and recurrent excitatory connectivity succumb to explosive run-away activity in response to external stimulation. We then show how STP effectively mitigates this instability by re-stabilizing ensemble dynamics in an inhibition-stabilized network (ISN) state, but only after generating a pronounced stimulus-triggered onset transient. We call this mechanism NTA and show that it yields selective onset responses that carry more relevant stimulus information than the subsequent steady-state. Finally, we characterize the functional benefits of inhibitory co-tuning, a feature that is widely observed in the brain (Wehr and Zador, 2003; Froemke et al., 2007; Okun and Lampl, 2008; Rupprecht and Friedrich, 2018) and readily emerges in computational models endowed with activity-dependent plasticity of inhibitory synapses (Vogels et al., 2011). We find that co-tuning prevents persistent attractor states but does not preclude NTA from occurring. Importantly, NTA purports that, following transient amplification, neuronal ensembles settle into a stable ISN state, consistent with recent studies suggesting that inhibition stabilization is a ubiquitous feature of cortical networks (Sanzeni et al., 2020). In summary, our work indicates that NTA is ideally suited to amplify stimuli rapidly through the interaction of strong recurrent excitation with STP.

Results

To understand the emergence of transient responses in recurrent neural networks, we studied rate-based population models with a supralinear, power law input-output function (Figure 1A and B; Ahmadian et al., 2013; Hennequin et al., 2018), which captures essential aspects of neuronal activation (Priebe et al., 2004), while also being analytically tractable. We first considered an isolated neuronal ensemble consisting of one excitatory (E) and one inhibitory (I) population (Figure 1A).

Figure 1 with 1 supplement see all

Download asset Open asset

Neuronal ensembles nonlinearly amplify inputs above a critical threshold.

(A) Schematic of the recurrent ensemble model consisting of an excitatory (blue) and an inhibitory population (red). (B) Supralinear input-output function given by a rectified power law with exponent $α = 2$ . (C) Firing rates of the excitatory (blue) and inhibitory population (red) in response to external stimulation during the interval from 2 to 4 s (gray bar). The stimulation was implemented by temporarily increasing the input $g_{E}$ . (D) Phase portrait of the system before stimulation (left; C orange) and during stimulation (right; C green). (E) Characteristic function $F (z)$ for varying input strength $g_{E}$ . Note that the function loses its zero crossings, which correspond to fixed points of the system for increasing external input. (F) Heat map showing the evoked firing rate of the excitatory population for different parameter combinations $J_{E E}$ and $g_{E}$ . The gray region corresponds to the parameter regime with unstable dynamics.

The dynamics of this network are given by

τ_{E} \frac{d r_{E}}{d t} = - r_{E} + {[J_{E E} r_{E} - J_{E I} r_{I} + g_{E}]}_{+}^{α_{E}},

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + {[J_{I E} r_{E} - J_{I I} r_{I} + g_{I}]}_{+}^{α_{I}},

where $r_{E}$ and $r_{I}$ are the firing rates of the excitatory and inhibitory population, $τ_{E}$ and $τ_{I}$ represent the corresponding time constants, $J_{X Y}$ denotes the synaptic strength from the population $Y$ to the population $X$ , where $X, Y \in {E, I}$ , $g_{E}$ and $g_{I}$ are the external inputs to the respective populations. Finally, $α_{E}$ and $α_{I}$ , the exponents of the respective input-output functions, are fixed at two unless mentioned otherwise. For ease of notation, we further define the weight matrix $J$ of the compound system as follows:

J = [\begin{matrix} J_{E E} & - J_{E I} \\ J_{I E} & - J_{I I} \end{matrix}] .

We were specifically interested in networks with strong recurrent excitation that can generate positive feedback dynamics in response to external inputs $g_{E}$ . Therefore, we studied networks with

d e t (J) = - J_{E E} J_{I I} + J_{I E} J_{E I} < 0 .

In contrast, networks in which recurrent excitation is met by strong feedback inhibition such that $d e t (J) > 0$ are unable to generate positive feedback dynamics provided that inhibition is fast enough (Ahmadian et al., 2013). Importantly, we assumed that most inhibition originates from recurrent connections (Franks et al., 2011; Large et al., 2016) and, hence, we kept the input to the inhibitory population $g_{I}$ fixed unless mentioned otherwise.

Nonlinear amplification of inputs above a critical threshold

We initialized the network in a stable low-activity state in the absence of external stimulation, consistent with spontaneous activity in cortical networks (Figure 1C). However, an input $g_{E}$ of sufficient strength, destabilized the network (Figure 1C). Importantly, this behavior is distinct from linear network models in which the network stability is independent of inputs (Materials and methods). The transition from stable to unstable dynamics can be understood by examining the phase portrait of the system (Figure 1D). Before stimulation, the system has a stable and an unstable fixed point (Figure 1D, left). However, both fixed points disappear for an input $g_{E}$ above a critical stimulus strength (Figure 1D, right).

To further understand the system’s bifurcation structure, we consider the characteristic function

F (z) = J_{E E} [z]_{+}^{α_{E}} - J_{E I} [det (J) \cdot J_{E I}^{- 1} [z]_{+}^{α_{E}} + J_{E I}^{- 1} J_{I I} z - J_{E I}^{- 1} J_{I I} g_{E} + g_{I}]_{+}^{α_{I}} - z + g_{E},

where $z$ denotes the total current into the excitatory population and $det (J)$ represents the determinant of the weight matrix (Kraynyukova and Tchumatchenko, 2018; Materials and methods). The characteristic function reduces the original two-dimensional system to one dimension, whereby the zero crossings of the characteristic function correspond to the fixed points of the original system (Eq. (1)-(2)). We use this correspondence to visualize how the fixed points of the system change with the input $g_{E}$ . Increasing $g_{E}$ shifts $F (z)$ upwards, which eventually leads to all zero crossings disappearing and the ensuing unstable dynamics (Figure 1E; Materials and methods). Importantly, for any weight matrix $J$ with negative determinant, there exists a critical input $g_{E}$ at which all fixed points disappear (Materials and methods). While for weak recurrent E-to-E connection strength $J_{E E}$ , the transition from stable dynamics to unstable is gradual, in that it happens at higher firing rates (Figure 1F), it becomes more abrupt for stronger $J_{E E}$ . Thus, our analysis demonstrates that individual neuronal ensembles with negative determinant $det (J)$ nonlinearly amplify inputs above a critical threshold by switching from initially stable to unstable dynamics.

Short-term plasticity, but not spike-frequency adaptation, can re-stabilize ensemble dynamics

Since unstable dynamics are not observed in neurobiology, we wondered whether neuronal spike frequency adaptation (SFA) or STP could re-stabilize the ensemble dynamics while keeping the nonlinear amplification character of the system. Specifically, we considered SFA of excitatory neurons, E-to-E short-term depression (STD), and E-to-I short-term facilitation (STF). We focused on these particular mechanisms because they are ubiquitously observed in the brain. Most pyramidal cells exhibit SFA (Barkai and Hasselmo, 1994) and most synapses show some form of STP (Markram et al., 1998; Zucker and Regehr, 2002; Pala and Petersen, 2015). Moreover, the time scales of these mechanisms are well-matched to typical timescales of perception, ranging from milliseconds to seconds (Tsodyks and Markram, 1997; Fairhall et al., 2001; Pozzorini et al., 2013).

When we simulated our model with SFA (Eqs. (21)–(23)), we observed different network behaviors depending on the adaptation strength. When adaptation strength was weak, SFA was unable to stabilize run-away excitation (Figure 2A; Materials and methods). Increasing the adaptation strength eventually prevented run-away excitation, but to give way to oscillatory ensemble activity (Figure 2—figure supplement 1). Finally, we confirmed analytically that SFA cannot stabilize excitatory run-away dynamics at a stable fixed point (Materials and methods). In particular, while the input is present, strong SFA creates a stable limit cycle with associated oscillatory ensemble activity (Figure 2—figure supplement 1; Materials and methods), which was also shown in previous modeling studies (van Vreeswijk and Hansel, 2001), but is not typically observed in sensory systems (DeWeese et al., 2003; Rupprecht and Friedrich, 2018).

Figure 2 with 12 supplements see all

Download asset Open asset

Short-term plasticity, but not spike-frequency adaptation, re-stabilizes ensemble dynamics.

(A) Firing rates of the excitatory (blue) and inhibitory population (red) in the presence of spike-frequency adaptation (SFA). During stimulation (gray bar) additional input is injected into the excitatory population. The inset shows a cartoon of how SFA affects spiking neuronal dynamics in response to a step current input. (B) Left: Same as (A) but in the presence of E-to-E short-term depression (STD). Right: Same as left but inactivating inhibition in the period marked in purple. (C) 3D plot of the excitatory activity $r_{E}$ , inhibitory activity $r_{I}$ , and the STD variable $x$ of the network in B left. The orange and green points mark the fixed points before/after and during stimulation. (D) Characteristic function $F (z)$ in networks with E-to-E STD. Different brightness levels correspond to different time points in B left. (E) Same as (B) but in the presence of E-to-I short-term facilitation (STF). (F) Inhibition-stabilized network (ISN) index, which corresponds to the largest real part of the eigenvalues of the Jacobian matrix of the E-E subnetwork with STD, as a function of time for the network with E-to-E STD in B left. For values above zero (dashed line), the ensemble is an ISN. (G) Analytical solution of non-ISN (magenta), ISN (green), paradoxical, and non-paradoxical regions for different parameter combinations $J_{E E}$ and the STD variable $x$ . The solid line separates the non-ISN and ISN regions, whereas the dashed line separates the non-paradoxical and paradoxical regions. (H) The normalized firing rates of the excitatory (blue) and inhibitory population (red) when injecting additional excitatory current into the inhibitory population before stimulation (left; orange bar in B), and during stimulation (right; green bar in B). Initially, the ensemble is in the non-ISN regime and injecting excitatory current into the inhibitory population increases its firing rate. During stimulation, however, the ensemble is an ISN. In this case, excitatory current injection into the inhibitory population results in a reduction of its firing rate, also known as the *paradoxical effect*.

Next, we considered STP, which is capable of saturating the effective neuronal input-output function (Mongillo et al., 2012; Zenke et al., 2015; Eqs. (37)–(39), Eqs. (41)–(43)). We first analyzed the stimulus-evoked network dynamics when we added STD to the recurrent E-to-E connections. Strong depression of synaptic efficacy resulted in a brief onset transient after which the ensemble dynamics quickly settled into a stimulus-evoked steady-state with slightly higher activity than the baseline (Figure 2B, left). After stimulus removal, the ensemble activity returned back to its baseline level (Figure 2B, left; Figure 2C). Notably, the ensemble dynamics settled at a stable steady state with a much higher firing rate, when inhibition was inactivated during stimulus presentation (Figure 2B, right). This shows that STP is capable of creating a stable high-activity fixed point, which is fundamentally different from the SFA dynamics discussed above. This difference in ensemble dynamics can be readily understood by analyzing the stability of the three-dimensional dynamical system (Materials and methods). We can gain a more intuitive understanding by considering self-consistent solutions of the characteristic function $F (z)$ . Initially, the ensemble is at the stable low activity fixed point. But the stimulus causes this fixed point to disappear, thus giving way to positive feedback which creates the leading edge of the onset transient (Figure 2B). However, because E-to-E synaptic transmission is rapidly reduced by STD, the curvature of $F (z)$ changes and a stable fixed point is created, thereby allowing excitatory run-away dynamics to terminate and the ensemble dynamics settle into a steady-state at low activity levels (Figure 2D). We found that E-to-I STF leads to similar dynamics (Figure 2E, left; Appendix 1) with the only difference that this configuration requires inhibition for network stability (Figure 2E, right), whereas E-to-E STD stabilizes activity even without inhibition, albeit at physiologically implausibly high activity levels. Importantly, the re-stabilization through either form of STP did not impair an ensemble’s ability to amplify stimuli during the initial onset phase.

Crucially, transient amplification in supralinear networks with STP occurs above a critical threshold (Figure 2—figure supplement 2), and requires recurrent excitation $J_{EE}$ to be sufficiently strong (Figure 2—figure supplement 2C, D). To quantify the amplification ability of these networks, we calculated the ratio of the evoked peak firing rate to the input strength, henceforth called the ‘Amplification index’. We found that amplification in STP-stabilized supralinear networks can be orders of magnitude larger than in linear networks with equivalent weights and comparable stabilized supralinear networks (SSNs) without STP (Figure 2—figure supplement 3). We stress that the resulting firing rates are parameter-dependent (Figure 2—figure supplement 4) and their absolute value can be high due to the high temporal precision of the onset peak and its short duration. In experiments, such high rates manifest themselves as precisely time-locked spikes with millisecond resolution (DeWeese et al., 2003; Wehr and Zador, 2003; Bolding and Franks, 2018; Gjoni et al., 2018).

Recent studies suggest that cortical networks operate as inhibition-stabilized networks (ISNs) (Sanzeni et al., 2020; Sadeh and Clopath, 2021), in which the excitatory network is unstable in the absence of feedback inhibition (Tsodyks et al., 1997; Ozeki et al., 2009). To that end, we investigated how ensemble re-stabilization relates to the network operating regime at baseline and during stimulation. Whether a network is an ISN or not is mathematically determined by the real part of the leading eigenvalue of the Jacobian of the excitatory-to-excitatory subnetwork (Tsodyks et al., 1997). We computed the leading eigenvalue in our model incorporating STP and referred to it as ‘ISN index’ (Materials and methods; Appendix 2). We found that in networks with STP the ISN index can switch sign from negative to positive during external stimulation, indicating that the ensemble can transition from a non-ISN to an ISN (Figure 2F). Notably, this behavior is distinct from linear network models in which the network operating regime is independent of the input (Materials and methods). Whether this switch between non-ISN to ISN occurred, however, was parameter dependent and we also found network configurations that were already in the ISN regime at baseline and remained ISNs during stimulation (Figure 2—figure supplement 5). Thus, re-stabilization was largely unaffected by the network state and consistent with experimentally observed ISN states (Sanzeni et al., 2020).

Theoretical studies have shown that one defining characteristic of ISNs in static excitatory and inhibitory networks is that injecting excitatory (inhibitory) current into inhibitory neurons decreases (increases) inhibitory firing rates, which is also known as the paradoxical effect (Tsodyks et al., 1997; Miller and Palmigiano, 2020). Yet, it is unclear whether in networks with STP, inhibitory stabilization implies paradoxical response and vice versa. We therefore analyzed the condition of being an ISN and the condition of having paradoxical response in networks with STP (Materials and methods; Appendix 2; Appendix 3). Interestingly, we found that in networks with E-to-E STD, the paradoxical effect implies inhibitory stabilization, whereas inhibitory stabilization does not necessarily imply paradoxical response (Figure 2G; Materials and methods), suggesting that having paradoxical effect is a sufficient but not necessary condition for being an ISN. In contrast, in networks with E-to-I STF, inhibitory stabilization and paradoxical effect imply each other (Appendix 2; Appendix 3). Therefore, paradoxical effect can be exploited as a proxy for inhibition stabilization for networks with STP we considered here. By injecting excitatory current into the inhibitory population, we found that the network did not exhibit the paradoxical effect before stimulation (Figure 2H, left; Figure 2—figure supplement 6). In contrast, injecting excitatory inputs into the inhibitory population during stimulation reduced their activity (Figure 2H, right; Figure 2—figure supplement 6). As demonstrated in our analysis, non-paradoxical response does not imply non-ISN (Figure 2—figure supplement 7; Materials and methods). We therefore examined the inhibition stabilization property of the ensemble by probing the ensemble behavior when a small transient perturbation to excitatory population activity is introduced while inhibition is frozen before stimulation and during stimulation. Before stimulation, the firing rate of the excitatory population slightly increases and then returns to its baseline after the transient perturbation (Figure 2—figure supplement 8). During stimulation, however, the transient perturbation leads to a transient explosion of the excitatory firing rate (Figure 2—figure supplement 8). These results further confirm that the ensemble shown in our example is initially a non-ISN before stimulation and can transition to an ISN with stimulation. By elevating the input level at the baseline in the model, the ensemble can be initially an ISN (Figure 2—figure supplement 5), resembling recent studies revealing that cortical circuits in the mouse V1 operate as ISNs in the absence of sensory stimulation (Sanzeni et al., 2020).

Despite the fact that the supralinear input-output function of our framework captures some aspects of intracellular recordings (Priebe et al., 2004), it is unbounded and thus allows infinitely high firing rates. This is in contrast to neurobiology where firing rates are bounded due to neuronal refractory effects. While this assumption permitted us to analytically study the system and therefore to gain a deeper understanding of the underlying ensemble dynamics, we wondered whether our main conclusions were also valid when we limited the maximum firing rates. To that end, we carried out the same simulations while capping the firing rate at 300 Hz. In the absence of additional SFA or STP mechanisms, the firing rate saturation introduced a stable high-activity state in the ensemble dynamics which replaced the unstable dynamics in the uncapped model. As above, the ensemble entered this high-activity steady-state when stimulated with an external input above a critical threshold and exhibited persistent activity after stimulus removal (Figure 2—figure supplement 9). While weak SFA did not change this behavior, strong SFA resulted in oscillatory behavior during stimulation consistent with previous analytical work (Figure 2—figure supplement 9, van Vreeswijk and Hansel, 2001), but did not in stable steady-states commonly observed in biological circuits. In the presence of E-to-E STD or E-to-I STF, however, the ensemble exhibited transient evoked activity at stimulation onset that was comparable to the uncapped case. Importantly, the ensemble did not show persistent activity after the stimulation (Figure 2—figure supplement 9). Finally, we confirmed that all of these findings were qualitatively similar in a realistic spiking neural network model (Figure 2—figure supplement 10; Materials and methods).

In summary, we found that neuronal ensembles can rapidly, nonlinearly, and transiently amplify inputs by briefly switching from stable to unstable dynamics before being re-stabilized through STP mechanisms. We call this mechanism nonlinear transient amplification (NTA) which, in contrast to balanced amplification (Murphy and Miller, 2009; Hennequin et al., 2012), arises from population dynamics with supralinear neuronal activation functions interacting with STP. While we acknowledge that there may be other nonlinear transient amplification mechanisms, in this article we restrict our analysis to the definition above. NTA is characterized by a large onset response, a subsequent ISN steady-state while the stimulus persists, and a return to a unique baseline activity state after the stimulus is removed. Thus, NTA is ideally suited to rapidly and nonlinearly amplify sensory inputs through recurrent excitation, like reported experimentally (Ko et al., 2011; Cossell et al., 2015), while avoiding persistent activity.

Co-tuned inhibition broadens the parameter regime of NTA in the absence of persistent activity

Up to now, we have focused on a single neuronal ensemble. However, to process information in the brain, several ensembles with different stimulus selectivity presumably coexist and interact in the same circuit. This coexistence creates potential problems. It can lead to multi-stable persistent attractor dynamics, which are not commonly observed and could have adverse effects on the processing of subsequent stimuli. One solution to this issue could be EI co-tuning, which arises in network models with plastic inhibitory synapses (Vogels et al., 2011) and has been observed experimentally in several sensory systems (Wehr and Zador, 2003; Froemke et al., 2007; Okun and Lampl, 2008; Rupprecht and Friedrich, 2018).

To characterize the conditions under which neuronal ensembles nonlinearly amplify stimuli without persistent activity, we analyzed the case of two interacting ensembles. More specifically, we considered networks with two excitatory ensembles and distinguished between global and co-tuned inhibition (Figure 3A). In the case of global inhibition, one inhibitory population non-specifically inhibits both excitatory populations (Figure 3A, left). In contrast, in networks with co-tuned inhibition, each ensemble is formed by a dedicated pair of an excitatory and an inhibitory population which can have cross-over connections, for instance, due to overlapping ensembles (Figure 3A, right).

Figure 3 with 1 supplement see all

Download asset Open asset

Co-tuned inhibition broadens the parameter regime of NTA in the absence of persistent activity.

(A) Schematic of two neuronal ensembles with global inhibition (left) and with co-tuned inhibition (right). (B) Firing rate dynamics of bi/multi-stable ensemble dynamics (left) and uni-stable (right). In both cases, additional excitatory inputs are injected into excitatory ensemble E1 during the period marked in gray. (C) Analytical solution of uni- and bi/multi-stability regions for global inhibition (left) and co-tuned inhibition (right). Co-tuning results in a larger parameter regime of uni-stability. The triangles correspond to the two examples in B.

Global inhibition supports winner-take-all competition and is therefore often associated with multi-stable attractor dynamics (Wong and Wang, 2006; Mongillo et al., 2008). We first illustrated this effect in a network model with global inhibition. When the recurrent excitatory connections within each ensemble were sufficiently strong, small amounts of noise in the initial condition led to one of the ensembles spontaneously activating at elevated firing rates, while the other ensemble’s activity remained low (Figure 3B, left). A specific external stimulation could trigger a switch from one state to the other in which the other ensemble was active at a high firing rate. Importantly, this change persisted even after the stimulus had been removed, a hallmark of multi-stable dynamics. In contrast, uni-stable systems have a global symmetric state in which both ensembles have the same activity in the absence of stimulation. While the stimulated ensemble showed elevated firing rates in response to the stimulus, its activity returned to the baseline level after the stimulus is removed (Figure 3B, right), consistent with experimental observations (DeWeese et al., 2003; Rupprecht and Friedrich, 2018; Bolding and Franks, 2018). Note that the only difference between these two models is that $J_{E E}$ is larger in the multi-stable example than in the uni-stable one.

Symmetric baseline activity is most consistent with activity observed in sensory areas. Hence, we sought to understand which inhibitory connectivity would be most conducive to maintain it. To that end, we analytically identified the uni-stability conditions, which are determined by the leading eigenvalue of the Jacobian matrix of the system, for networks with varying degrees of EI co-tuning (Materials and methods). We found that a broader parameter regime underlies uni-stability in networks with co-tuned inhibition than global inhibition (Figure 3C). Notably, this conclusion is general and extends to networks with an arbitrary number of ensembles (Materials and methods). In comparison to the ensemble with global inhibition, the ensemble with co-tuned inhibition exhibits weaker — but still strong — NTA (Figure 3—figure supplement 1). Thus, co-tuned inhibition broadens the parameter regime in which NTA is possible while simultaneously avoiding persistent attractor dynamics.

NTA provides better pattern completion than fixed points while retaining stimulus selectivity

Neural circuits are capable of generating stereotypical activity patterns in response to partial cues and forming distinct representations in response to different stimuli (Carrillo-Reid et al., 2016; Marshel et al., 2019; Bolding et al., 2020; Vinje and Gallant, 2000; Cayco-Gajic and Silver, 2019). To test whether NTA achieves pattern completion while retaining stimulus selectivity, we analyzed the transient onset activity in our models and compared it to the fixed point activity.

To investigate pattern completion and stimulus selectivity in our model, we considered a co-tuned network with E-to-E STD and two distinct excitatory ensembles $E 1$ and $E 2$ . We gave additional input $g_{E 1}$ to a Subset 1, consisting of 75% of the neurons in ensemble $E 1$ (Figure 4A). We then measured the evoked activity in the remaining 25% of the excitatory neurons in $E 1$ to quantify pattern completion. To assess stimulus selectivity, we injected additional input $g_{E 1}$ into the entire $E 1$ ensemble during the second stimulation phase (Figure 4A) while measuring the activity of $E 2$ . We found that neurons in Subset 2, which did not receive additional input, showed large onset responses, their steady-state activity was largely suppressed (Figure 4B). Despite the fact that inputs to $E 1$ caused increased transient onset responses in $E 2$ , the amount of increase was orders of magnitude smaller than in $E 1$ (Figure 4B). To quantify pattern completion, we defined the

Association index = 1 + \frac{r_{E 1_{2}} - r_{E 1_{1}}}{r_{E 1_{2}} + r_{E 1_{1}}} .

Figure 4 with 2 supplements see all

Download asset Open asset

NTA yields stronger pattern completion than fixed points while retaining stimulus selectivity.

(A) Schematic of the network setup used to probe pattern completion and stimulus selectivity. To assess the effect on pattern completion, 75% of the neurons (Subset 1) in ensemble E1 received additional input g_E1 during Phase one (2–4 s), while we recorded the firing rate of the remaining 25% (Subset 2) in the excitatory ensemble E1. To evaluate the impact on stimulus selectivity, all neurons in E1 received additional inputs g_E1 in Phase two (6–8 s) while the firing rate of E2 was measured. A downstream neuron’s ability to discriminate between E1 or E2 being active depends on whether their activity is well separated by a symmetric decision boundary (inset). (B) Examples of firing rates of Subset 2 of E1 (left, blue) and E2 (right, green) with E-to-E STD. (C) Association index as a function of input g_E1 for the onset peak amplitude (magenta solid line) and fixed point activity (gray dashed line) for E-to-E STD. (D) Distance to the decision boundary (see panel A, inset) as a function of input g_E1 for the onset peak amplitude (magenta solid line) and fixed point activity (gray dashed line) for E-to-E STD. (**E and F**) Same as C and D but as a function of β, which controls the inner- and inter-ensemble connection strength.

Here, $r_{E 1_{1}}$ and $r_{E 1_{2}}$ correspond to the subpopulation activities of $E 1$ , respectively. By definition, the Association index ranges from zero to one, with larger values indicating stronger associativity. In addition, to quantify the selectivity between $E 1$ and $E 2$ , we considered a symmetric binary classifier (Figure 4A, inset) and measured the distance to the decision boundary (Materials and methods). Note that the Association index was computed during Phase one and the distance to the decision boundary during Phase two in this simulation paradigm (Figure 4B).

With these definitions, we ran simulations with different input strengths $g_{E 1}$ . We found that the onset peaks showed stronger association than the fixed point activity (Figure 4C). Note that the Association index at the fixed point remained zero, a direct consequence of $r_{E 1_{2}}$ being suppressed to zero. Furthermore, we found that the distance between the transient onset response and the decision boundary was always greater than for the fixed point activity (Figure 4D) showing that onset responses retain stimulus selectivity. While the fixed point activity of the unstimulated co-tuned neurons is zero in the given example, stimulating a subset of neurons in one ensemble can lead to an increase in the fixed point activity of the unstimulated neurons in the same ensemble under certain conditions (Figure 4—figure supplement 1; Appendix 4), which is consistent with pattern completion experiments (Carrillo-Reid et al., 2016; Marshel et al., 2019) showing that unstimulated neurons from the same ensemble can remain active throughout the whole stimulation period.

To investigate how the recurrent excitatory connectivity affects both pattern completion and stimulus selectivity, we introduced the parameter $β$ which controls recurrent excitatory tuning by trading off within-ensemble E-to-E strength $J_{E E}$ relative to the inter-ensemble strength $J_{E E}^{^{'}}$ (Figure 4A) such that $J_{E E} = β J_{tot}$ and $J_{E E}^{^{'}} = (1 - β) J_{tot}$ . These definitions ensure that the total weight $J_{tot} = J_{E E} + J_{E E}^{^{'}}$ remains constant for any choice of $β$ . Notably, the overall recurrent excitation strength within an ensemble $J_{E E}$ increases with increasing $β$ . When $β$ is larger than 0.5, the excitatory connection strength within the ensemble $J_{E E}$ exceeds the one between ensembles $J_{E E}^{^{'}}$ .

We found that pattern completion ability monotonically increases with $β$ with a pronounced onset for $β §gt; 0.6$ where NTA takes hold (Figure 4E). Moreover, in this regime the two stimulus representations are well separated (Figure 4F) which ensures stimulus selectivity also during onset transients. Together, these findings recapitulate the point that recurrent excitatory tuning is a key determinant of network dynamics. Finally, we confirmed that our findings were also valid in networks with E-to-I STF (Figure 4—figure supplement 2), which is commonly observed in the brain (Markram et al., 1998; Zucker and Regehr, 2002; Pala and Petersen, 2015). In summary, NTA’s transient onset responses maintain stimulus selectivity and result in overall better pattern completion than fixed point activity.

NTA provides higher amplification and pattern separation in morphing experiments

So far, we only considered input to one ensemble. To examine how representations in our model are affected by ambiguous inputs to several ensembles, we performed additional morphing experiments (Freedman et al., 2001; Niessing and Friedrich, 2010). To that end, we introduced the parameter $p$ which interpolates between two input stimuli which target $E 1$ and $E 2$ respectively. When $p$ is zero, all additional input is injected into $E 1$ . For $p$ equal to one, all additional input is injected into $E 2$ . Finally, $p$ equal to 0.5 corresponds to the symmetric case in which $E 1$ and $E 2$ receive the same amount of additional input (Figure 5A).

Figure 5 with 2 supplements see all

Download asset Open asset

NTA provides stronger amplification and pattern separation in morphing experiments than fixed point activity.

(A) Schematic of the morphing stimulation paradigm. The fraction of the additional inputs into the two excitatory ensembles is controlled by the parameter p. (B) Peak amplitude of E1 (blue) and E2 (green) as a function of p for E-to-E STD. Brightness levels represent different recurrent E-to-E connection strengths J_EE . (C) Same as in B but for fixed point activity. (D) Distance to the decision boundary as a function of p for the peak onset response (magenta solid line) and fixed point activity (gray dashed line) for E-to-E STD in a network with J’_IE = 0.4. (E) Same as D but with different E-to-I connection strengths J’_IE across ensembles for a network with J_EE = 1.2.

First, we investigated how the recurrent excitatory connection strength within each ensemble $J_{E E}$ affects the onset peak amplitude and fixed point activity. We found that the peak amplitudes depend strongly on $J_{E E}$ , whereas the fixed point activity was only weakly dependent on $J_{E E}$ (Figure 5B and C). When we disconnected the ensembles by completely eliminating all recurrent excitatory connections, activity was noticeably decreased (Figure 5B and C). This illustrates, that recurrent excitation does play an important role in selectively amplifying specific stimuli similar to experimental observations (Marshel et al., 2019; Peron et al., 2020), but that amplification is highest at the onset.

Further, we examined the impact of competition through lateral inhibition as a function of the E-to-I inter-ensemble strength $J_{I E}^{^{'}}$ (Materials and methods). As above, we quantified its impact by measuring the representational distance to the decision boundary for the transient onset responses and fixed point activity. We found that regardless of the specific STP mechanism, the distance was larger for the onset responses than for the fixed point activity, consistent with the notion that the onset dynamics separate stimulus identity reliably (Figure 5D and E). Since the absolute activity levels between onset and fixed point differed substantially, we further computed the relative pattern Separation index $(r_{E 2} - r_{E 1}) / (r_{E 1} + r_{E 2})$ and found that the onset transient provides better pattern separation ability for ambiguous stimuli with $p$ close to 0.5 (Figure 5—figure supplement 1) provided that the E-to-I connection strength across ensembles $J_{I E}^{^{'}}$ is strong enough. All the while separability for the onset transient was slightly decreased for distinct inputs with $p \in {0, 1}$ in comparison to the fixed point. In contrast, fixed points clearly separated such pure stimuli while providing weaker pattern separation for ambiguous input combinations. Importantly, these findings qualitatively held for networks with NTA mediated by E-to-I STF (Figure 5—figure supplement 2). Thus, NTA provides stronger amplification and pattern separation than fixed point activity in response to ambiguous stimuli.

NTA in spiking neural networks

Thus far, our analysis relied on power law neuronal input-output functions in the interest of analytical tractability. To test whether our findings also qualitatively apply to more realistic network models, we built a spiking neural network consisting of randomly connected 800 excitatory and 200 inhibitory neurons, in which the E-to-E synaptic connections were subject to STD (Materials and methods). Here, we defined five overlapping ensembles, each corresponding to 200 randomly selected excitatory neurons. During an initial simulation phase (0–22 s), we consecutively stimulated each ensemble by giving additional input to their excitatory neurons, whereas the input to other neurons remained unchanged (Figure 6A). In addition, we also tested pattern completion by stimulating only 75% (Subset 1) of the neurons belonging to Ensemble 5 (22–24 s; Figure 6A). We quantified each ensemble’s activity by calculating the population firing rate of the ensemble (Materials and methods). As in the case of the rate-based model, the neuronal ensembles in the spiking model generated pronounced transient onset responses. We then measured the difference of peak ensemble activity and fixed point activity between the stimulated ensemble and the remaining unstimulated ensembles (Materials and methods). As for the rate-based networks, this difference was consistently larger for the onset peak than for the fixed point (Figure 6B and C). Thus, transient onset responses allow better stimulus separation than fixed points also in spiking neural network models.

Figure 6 with 1 supplement see all

Download asset Open asset

Spiking neural network simulations qualitatively reproduce NTA dynamics of rate models.

(A) Spiking activity of excitatory (blue) and inhibitory (red) neurons in a spiking neural network. From 2 to 20 s, Ensembles 1–5 individually received additional input for 2 s each (colored bars). From 22 to 24 s, 75% of Ensemble 5 neurons (Subset 1) received additional input, whereas the rest 25% of Ensemble 5 neurons (Subset 2) did not receive additional input. The symbols at the top designate the different simulation phases of baseline activity, the onset transients, and the fixed point activity. Different colors correspond to the distinct stimulation periods. (B) Ensemble activity (colors). (C) Difference in ensemble activity between the stimulated ensemble with the remaining ensembles for the transient onset peak and the fixed point. Points correspond to the different stimulation periods. (D) Spiking activity during the interval 0–10 s represented in the PCA basis spanned by the first two principal components which captured approximately 40% of the total variance. The colored lines represent the PC trajectories of the first two stimuli shown in A and B. Triangles, points and crosses correspond to the onset peak, fixed point, and baseline activity, respectively. (E) Ensemble activity of Subset 1 (purple) and Subset 2 (gray) of Ensemble 5 from 16 to 26 s. Onset peaks are marked by triangles.

Finally, to visualize the neural activity, we projected the binned spiking activity during the first 10 s of our simulation onto its first two principal components. Notably, the PC trajectory does not exhibit a pronounced rotational component (Figure 6D) as activity is confined to one specific ensemble, consistent with experiments (Marshel et al., 2019). Furthermore, we computed the fifth ensemble’s activity for Subset 1 and 2 during the time interval 16–26 s. In agreement with our rate models, neurons in Subset 2 which did not receive additional inputs showed a strong response at the onset (Figure 6E), but not at the fixed point, suggesting that the strongest pattern completion occurs during the initial amplification phase. Finally, we also observed higher-than-baseline fixed point activity in unstimulated neurons of Subset 2 in spiking neural networks (Figure 6—figure supplement 1). Thus, the key characteristics of NTA are preserved across rate-based and more realistic spiking neural network models.

Discussion

In this study, we demonstrated that neuronal ensemble models with recurrent excitation and suitable forms of STP exhibit nonlinear transient amplification (NTA), a putative mechanism underlying selective amplification in recurrent circuits. NTA combines a supralinear neuronal transfer function, recurrent excitation between neurons with similar tuning, and pronounced STP. Using analytical and numerical methods, we showed that NTA generates rapid transient onset responses during which optimal stimulus separation occurs rather than at steady-states. Additionally, we showed that co-tuned inhibition is conducive to prevent the emergence of persistent activity, which could otherwise interfere with processing subsequent stimuli. In contrast to balanced amplification (Murphy and Miller, 2009), NTA is an intrinsically nonlinear mechanism for which only stimuli above a critical threshold are amplified effectively. While the precise threshold value is parameter-dependent, it can be arbitrarily low provided the excitatory recurrent connections are sufficiently strong (Figure 1F). Importantly, such a critical activation threshold offers a possible explanation for sensory perception experiments which show similar threshold behavior (Marshel et al., 2019; Peron et al., 2020). Following transient amplification, ensemble dynamics are inhibition-stabilized, which renders our model compatible with existing work on SSNs (Ahmadian et al., 2013; Rubin et al., 2015; Hennequin et al., 2018; Kraynyukova and Tchumatchenko, 2018; Echeveste et al., 2020). Thus, NTA provides a parsimonious explanation for why sensory systems may rely upon neuronal ensembles with recurrent excitation in combination with EI co-tuning, and pronounced STP dynamics.

Several theoretical studies approached the problem of transient amplification in recurrent neural network models. Loebel and Tsodyks, 2002 have described an NTA-like mechanism as a driver for powerful ensemble synchronization in rate-based networks and in spiking neural network models of auditory cortex (Loebel et al., 2007). Here, we generalized this work to both E-to-E STD and E-to-I STF and provide an in-depth characterization of its amplification capabilities, pattern completion properties, and the resulting network states with regard to their inhibition-stabilization properties. Moreover, we showed that SFA cannot provide similar network stabilization and explored how EI co-tuning interacts with NTA. Finally, we contrasted NTA to alternative transient amplification mechanisms. Balanced amplification is a particularly well-studied transient amplification mechanism (Murphy and Miller, 2009; Goldman, 2009; Hennequin et al., 2014; Bondanelli and Ostojic, 2020; Gillett et al., 2020; Christodoulou et al., 2021) that relies on non-normality of the connectivity matrix to selectively and rapidly amplify stimuli. Importantly, balanced amplification occurs in networks in which strong recurrent excitation is appropriately balanced by strong recurrent inhibition. It is capable of generating rich transient activity in linear network models (Hennequin et al., 2014), and selectively amplifies specific activity patterns, but without a specific activation threshold. In addition, in spiking neural networks, strong input can induce synchronous firing at the population level which is subsequently stabilized by strong feedback inhibition without the requirement for STP mechanisms (Stern et al., 2018). These properties contrast with NTA, which has a nonlinear activation threshold and intrinsically relies on STP to stabilize otherwise unstable run-away dynamics. Due to the switch of the network’s dynamical state, NTA’s amplification can be orders of magnitudes larger than balanced amplification (Figure 2—figure supplement 3). Interestingly, after the transient amplification phase, ensemble dynamics settle in an inhibitory-stabilized state, which renders NTA compatible with previous work on SSNs but in the presence of STP. Finally, although NTA and balanced amplification rely on different amplification mechanisms, they are not mutually exclusive and could, in principle, co-exist in biological networks.

NTA’s requirement to generate positive feedback dynamics through recurrent excitation, motivated our focus on networks with $det (J) < 0$ . As demonstrated in previous work (Ahmadian et al., 2013), supralinear networks with $det (J) > 0$ and instantaneous inhibition ( $τ_{I} / τ_{E} \to 0$ ) are always stable for any given input, they are thus unable to generate positive feedback dynamics. In addition, networks with $det (J) > 0$ can exhibit a range of interesting behaviors, for example, oscillatory dynamics and persistent activity (Kraynyukova and Tchumatchenko, 2018). It is worth noting, however, that for delayed or slow inhibition, stimulation can still lead to unstable network dynamics in networks with $det (J) > 0$ . Nevertheless, our simulations suggest that our main conclusions about the stabilization mechanisms still hold (Figure 2—figure supplement 11).

NTA shares some properties with the notion of network criticality in the brain, like synchronous activation of cell ensembles (Plenz and Thiagarajan, 2007) and STP which can tune networks to a critical state (Levina et al., 2007). However, in contrast to most models of criticality, in NTA an ensemble briefly transitions to supercritical dynamics in a controlled, stimulus-dependent manner rather than spontaneously. Yet, how the two paradigms are connected at a more fundamental level, is an intriguing question left for future work. Furthermore, recurrent co-tuned inhibition is essential for NTA to ensure uni-stability and selectivity through the suppression of ensembles with different tuning. This requirement is similar in flavor to semi-balanced networks characterized by excess inhibition to some excitatory ensembles while others are balanced (Baker et al., 2020). However, the theory of semi-balanced networks has, so far, only been applied to steady-state dynamics while ignoring transients and STP. EI co-tuning prominently features in several models and was shown to support network stability (Vogels et al., 2011; Hennequin et al., 2017; Znamenskiy et al., 2018), efficient coding (Denève and Machens, 2016), novelty detection (Schulz et al., 2021), changes in neuronal variability (Hennequin et al., 2018; Rost et al., 2018), and correlation structure (Wu et al., 2020). Moreover, some studies have argued that EI balance and co-tuning could increase robustness to noise in the brain (Rubin et al., 2017). The present work mainly highlights its importance for preventing multi-stability and delay activity in circuits not requiring such long-timescale dynamics.

NTA is consistent with several experimental findings. First, our model recapitulates the key findings of Shew et al., 2015 who showed ex vivo that strong sensory inputs cause a transient shift to a supercritical state, after which adaptive changes rapidly tune the network to criticality. Second, NTA requires strong recurrent excitatory connectivity between neurons with similar tuning, which has been reported in experiments (Ko et al., 2011; Cossell et al., 2015; Peron et al., 2020). Third, ensemble activation in our model depends on a critical stimulus strength in line with recent all-optical experiments in the visual cortex, which further link ensemble activation with a perceptual threshold (Marshel et al., 2019). Fourth, sensory networks are uni-stable in that they return to a non-selective activity state after the removal of the stimulus and usually do not show persistent activity (DeWeese et al., 2003; Mazor and Laurent, 2005; Rupprecht and Friedrich, 2018). Fifth, our work shows that NTA’s onset responses encode stimulus identity better than the fixed point activity, consistent with experiments in the locust antennal lobe (Mazor and Laurent, 2005) and research supporting that the brain relies on coactivity on short timescales to represent information (Stopfer et al., 1997; Engel et al., 2001; Harris et al., 2003; El-Gaby et al., 2021). Yet, it remains to be seen whether these findings are also coherent with data on the temporal evolution in other sensory systems. Finally, EI co-tuning, which is conducive for NTA, has been found ubiquitously in different sensory circuits (Wehr and Zador, 2003; Froemke et al., 2007; Okun and Lampl, 2008; Rupprecht and Friedrich, 2018; Znamenskiy et al., 2018).

In our model, we made several simplifying assumptions. For instance, we kept the input to inhibitory neurons fixed and only varied the input to the excitatory population. This step was motivated by experiments in the piriform cortex where the total inhibition is dominated by feedback inhibition (Franks et al., 2011). Nevertheless, significant feedforward inhibition was observed in other areas (Bissière et al., 2003; Cruikshank et al., 2007; Ji et al., 2016; Miska et al., 2018). While an in-depth comparison for different origins of inhibition was beyond the scope of the present study, we found that increasing the inputs to the excitatory population and inhibitory population by the same amount can still lead to NTA (Figure 1—figure supplement 1; Figure 2—figure supplement 12; Materials and methods), suggesting that our main findings can remain unaffected in the presence of substantial feedforward inhibition. In addition, we limited our analysis to only a few overlapping ensembles. It will be interesting future work to study NTA in the case of many interacting and potentially overlapping ensembles and to determine the maximum storage capacity above which performance degrades. Finally, we anticipate that temporal differences in excitatory and inhibitory synaptic transmission may be important to preserve NTA’s stimuli selectivity.

Our model makes several predictions. In contrast to balanced amplification, in which the network operating regime depends solely on the connectivity, an ensemble involved in NTA can transition from a non-ISN to an ISN state. Such a transition is consistent with noise variability observed in sensory cortices (Hennequin et al., 2018) and could be tested experimentally by probing the paradoxical effect under different stimulation conditions (Figure 2G–H; Figure 2—figure supplement 6). Moreover, NTA predicts that onset activity provides a better stimulus encoding and its activity is correlated with the fixed point activity. This signature is different from purely non-normal amplification mechanisms which would involve a wave of neuronal activity across several distinct ensembles similar to a synfire chain (Abeles, 1991). The difference should be clearly discernible in data. Since NTA relies on recurrent excitation between ensemble neurons, it suggests normal dynamics in which distinct ensembles first activate and then inactivate. The resulting dynamics have weak rotational components (Figure 6D) as seen in some experiments (Marshel et al., 2019). Strong non-normal amplification, on the other hand, relies on sequential activation associated with pronounced rotational dynamics (Hennequin et al., 2014; Gillett et al., 2020), as for instance observed in motor areas (Churchland et al., 2012). Although both non-normal mechanisms and NTA are likely to co-exist in the brain, we speculate that strong NTA is best suited for, and thus most like to be found in, sensory systems.

In summary, we introduced a general theoretical framework of selective transient signal amplification in recurrent networks. Our approach derives from the minimal assumptions of a nonlinear neuronal transfer function, recurrent excitation within neuronal ensembles, and STP. Importantly, our analysis revealed the functional benefits of STP and EI co-tuning, both pervasively found in sensory circuits. Finally, our work suggests that transient onset responses rather than steady-state activity are ideally suited for coactivity-based stimulus encoding and provides several testable predictions.

Materials and methods

Stability conditions for supralinear networks

Request a detailed protocol

The dynamics of a neuronal ensemble consisting of one excitatory and one inhibitory population with a supralinear, power law input-output function can be described as follows:

τ_{E} \frac{d r_{E}}{d t} = - r_{E} + {[J_{E E} r_{E} - J_{E I} r_{I} + g_{E}]}_{+}^{α_{E}}

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + {[J_{I E} r_{E} - J_{I I} r_{I} + g_{I}]}_{+}^{α_{I}}

The Jacobian $M$ of the system is given by

M = [\begin{matrix} τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) & - τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} \\ τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} & - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) \end{matrix}]

To ensure that the system is stable, the product of $M$ ’s eigenvalues $λ_{1} λ_{2}$ , which is equivalent to the determinant of $M$ , has to be positive. In addition, the sum of the two eigenvalues $λ_{1} + λ_{2}$ , which corresponds to $tr (M)$ , has to be negative. We therefore obtained the following two stability conditions

λ_{1} λ_{2} = - τ_{E}^{- 1} τ_{I}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) + τ_{E}^{- 1} τ_{I}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} §gt; 0

λ_{1} + λ_{2} = τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) §lt; 0

Notably, the stability conditions depend on the firing rate of the excitatory population $r_{E}$ and the inhibitory population $r_{I}$ . Since firing rates are input-dependent, the stability of supralinear networks is input-dependent. In contrast, in linear networks in which $α_{E} = α_{I} = 1$ , the conditions can be simplified to

λ_{1} λ_{2} = - τ_{E}^{- 1} τ_{I}^{- 1} (J_{E E} - 1) (1 + J_{I I}) + τ_{E}^{- 1} τ_{I}^{- 1} J_{E I} J_{I E} §gt; 0

λ_{1} + λ_{2} = τ_{E}^{- 1} (J_{E E} - 1) - τ_{I}^{- 1} (1 + J_{I I}) §lt; 0

and are thus input-independent.

ISN index for supralinear networks

Request a detailed protocol

If an ensemble is unstable without feedback inhibition, then the ensemble is an ISN (Tsodyks et al., 1997). To determine whether a given system is an ISN, we analyzed the stability of the E-E subnetwork, which is determined by the real part of the leading eigenvalue of the Jacobian of the E-E subnetwork. In the following, we call this leading eigenvalue the ‘ISN index’, which is defined as follows:

ISN index = τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1)

A positive ISN index indicates the system is an ISN. Otherwise, the system is non-ISN. For supralinear networks in which $α_{E} §gt; 1$ , the ISN index depends on the firing rates, inputs can therefore switch the network from non-ISN to ISN. In contrast, $α_{E} = 1$ for linear networks which renders the ISN index firing rate independent.

Characteristic function

Request a detailed protocol

To investigate how network stability changes with input, we trace the steps of Kraynyukova and Tchumatchenko, 2018 and define the characteristic function $F (z)$ as follows:

F (z) = J_{E E} [z]_{+}^{α_{E}} - J_{E I} [det (J) \cdot J_{E I}^{- 1} [z]_{+}^{α_{E}} + J_{E I}^{- 1} J_{I I} z - J_{E I}^{- 1} J_{I I} g_{E} + g_{I}]_{+}^{α_{I}} - z + g_{E}

where

z = J_{E E} r_{E} - J_{E I} r_{I} + g_{E}

is the current into the excitatory population. The characteristic function simplifies the original two-dimensional system to a one-dimensional system, and the zero crossings of $F (z)$ correspond to the fixed points of the original system. For $z \geq 0$ , we note:

\frac{d F (z)}{d z} = J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - J_{E I} α_{I} (det (J) \cdot J_{E I}^{- 1} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} + J_{E I}^{- 1} J_{I I}) r_{I}^{\frac{α_{I} - 1}{α_{I}}} - 1 = - τ_{E} τ_{I} λ_{1} λ_{2}

Therefore, if the derivative of $F (z)$ evaluated at one of its roots is positive, the corresponding fixed point is a saddle point. Note that as $r_{E}$ and $r_{I}$ increase, the term in parenthesis becomes dominant. To ensure that $λ_{1} λ_{2}$ is negative also for large $r_{E}$ and $r_{I}$ , the determinant of the weight matrix $det (J)$ has to be positive. Therefore, $det (J)$ has a decisive impact on the curvature of $F (z)$ . In systems with negative determinant, $F (z)$ bends upwards for large $z$ . In contrast, $F (z)$ asymptotically bends downwards in systems with positive determinant. Hence, the high-activity steady-state of systems with negative determinant is unstable. In addition, we can simplify the above condition to the determinant of the weight matrix which is a necessary condition for network stability at any firing rate:

det (J) = - J_{E E} J_{I I} + J_{I E} J_{E I} > 0

To investigate how the network stability changes with input $g_{E}$ , we examined how $F (z)$ varies with changing input $g_{E}$ by calculating the derivative of $F (z)$ with respect to $g_{E}$ ,

\frac{d F (z)}{d g_{E}} = α_{I} J_{I I} [det (J) \cdot J_{E I}^{- 1} [z]_{+}^{α_{E}} + J_{E I}^{- 1} J_{I I} z - J_{E I}^{- 1} J_{I I} g_{E} + g_{I}]_{+}^{α_{I} - 1} + 1

Since $\frac{d F (z)}{d g_{E}}$ is positive, increasing $g_{E}$ always shifts $F (z)$ upwards, eventually leading to the vanishing of all roots and, thus, unstable dynamics in supralinear networks with negative $det (J)$ . In scenarios in which feedforward input to the inhibitory population also changes, we have

\begin{aligned} \frac{d F (z)}{d t} = & \frac{\partial F (z)}{\partial g_{E}} \frac{d g_{E}}{d t} + \frac{\partial F (z)}{\partial g_{I}} \frac{d g_{I}}{d t} \\ = & (α_{I} J_{I I} [det (J) \cdot J_{E I}^{- 1} [z]_{+}^{α_{E}} + J_{E I}^{- 1} J_{I I} z - J_{E I}^{- 1} J_{I I} g_{E} + g_{I}]_{+}^{α_{I} - 1} + 1) Δ g_{E} \\ - α_{I} J_{E I} [det (J) \cdot J_{E I}^{- 1} [z]_{+}^{α_{E}} + J_{E I}^{- 1} J_{I I} z - J_{E I}^{- 1} J_{I I} g_{E} + g_{I}]_{+}^{α_{I} - 1} Δ g_{I} \end{aligned}

When the change in stimulation strength into the excitatory ( $Δ g_{E}$ ) and the inhibitory population ( $Δ g_{I}$ ) are the same, $\frac{d F (z)}{d t}$ is always positive provided $J_{I I}$ is greater than $J_{E I}$ . Hence, depending on the value of $\frac{J_{I I}}{J_{E I}}$ , stimulation can lead to unstable network dynamics even when the input to the inhibitory population increases more than to the excitatory population.

Spike-frequency adaptation (SFA)

We modeled SFA of excitatory neurons as an activity-dependent negative feedback current (Benda and Herz, 2003; Brette and Gerstner, 2005):

τ_{E} \frac{d r_{E}}{d t} = - r_{E} + {[J_{E E} r_{E} - J_{E I} r_{I} + g_{E}]}_{+}^{α_{E}} - a

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + {[J_{I E} r_{E} - J_{I I} r_{I} + g_{I}]}_{+}^{α_{I}}

τ_{a} \frac{d a}{d t} = - a + b r_{E}

where $a$ is the adaptation variable, $τ_{a}$ is the adaptation time constant, and $b$ is the adaptation strength.

Stability conditions in networks with SFA

Request a detailed protocol

The Jacobian $M_{S F A}$ of the system with SFA is given by

M_{S F A} = [\begin{matrix} τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) & - τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} & - τ_{E}^{- 1} \\ τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} & - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) & 0 \\ τ_{a}^{- 1} b & 0 & - τ_{a}^{- 1} \end{matrix}]

The characteristic polynomial of the system with SFA can be written as follows (Horn and Johnson, 1985):

λ^{3} - tr (M_{S F A}) λ^{2} + (A_{11} + A_{22} + A_{33}) λ - det (M_{S F A}) = 0

where $tr (M_{S F A})$ and $det (M_{S F A})$ are the trace and the determinant of the Jacobian matrix $M_{S F A}$ , A₁₁, A₂₂, and A₃₃ are the matrix cofactors. More specifically,

tr (M_{S F A}) = τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) - τ_{a}^{- 1}

A_{11} = | \begin{matrix} - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) & 0 \\ 0 & - τ_{a}^{- 1} \end{matrix} | = τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) τ_{a}^{- 1}

A_{22} = | \begin{matrix} τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) & - τ_{E}^{- 1} \\ τ_{a}^{- 1} b & - τ_{a}^{- 1} \end{matrix} | = - τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{a}^{- 1} + τ_{a}^{- 1} b τ_{E}^{- 1}

\begin{aligned} A_{33} = & | \begin{array}{cc} τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) & - τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} \\ τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} & - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) \end{array} | \\ = & - τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) + τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} \end{aligned}

\begin{aligned} A_{11} + A_{22} + A_{33} = & τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) τ_{a}^{- 1} - τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{a}^{- 1} + τ_{a}^{- 1} b τ_{E}^{- 1} \\ - τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) + τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} \end{aligned}

\begin{aligned} det (M_{S F A}) = & τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) τ_{a}^{- 1} \\ - τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} τ_{a}^{- 1} - τ_{a}^{- 1} b τ_{E}^{- 1} τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) \end{aligned}

To ensure that the dynamics of the system are stable, the real parts of the eigenvalues of the Jacobian at the fixed point, and thus all roots of the characteristic polynomial have to be negative. Since the product of the roots is equal to $det (M_{S F A})$ , $- det (M_{S F A})$ has to be positive. We then have

b > \frac{α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} (J_{E E} - det (J) \cdot α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}})}{1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}} - 1

Since SFA does not modify the synaptic connections, the term $J_{E E} - det (J) \cdot α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}$ is positive for networks with $det (J) < 0$ .

In the large $r_{E}$ limit, if $b$ is small such that the above condition cannot be fulfilled, $det (M_{S F A})$ is then positive, suggesting that the Jacobian of the system has always at least one positive eigenvalue. Therefore, the dynamics of the system cannot be stabilized in the presence of small $b$ .

In addition, $A_{11} + A_{22} + A_{33}$ is equal to $λ_{1} λ_{2} + λ_{2} λ_{3} + λ_{1} λ_{3}$ , with the roots of the characteristic polynomial $λ_{1}$ , $λ_{2}$ , and $λ_{3}$ . If all roots are real and negative, $A_{11} + A_{22} + A_{33}$ has to be positive. If one root is real and negative and two other roots are complex conjugates, to ensure that all roots have negative real parts, one necessary condition is $A_{11} + A_{22} + A_{33} §gt; 0$ . From the $tr (M_{S F A})$ and $det (M_{S F A})$ conditions, we have

A_{11} + A_{22} + A_{33} §gt; τ_{a}^{- 1} (- τ_{a}^{- 1} + b τ_{E}^{- 1}) - b τ_{E}^{- 1} τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}})

As a result, if $τ_{a}^{- 1} (- τ_{a}^{- 1} + b τ_{E}^{- 1}) - b τ_{E}^{- 1} τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) §gt; 0$ , $A_{11} + A_{22} + A_{33}$ is guaranteed to be positive. We therefore have

b [τ_{a}^{- 1} τ_{E}^{- 1} - τ_{E}^{- 1} τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}})] §gt; τ_{a}^{- 2}

Note that $τ_{a}$ has to be small, in other words, SFA has to be fast, so that $τ_{a}^{- 1} τ_{E}^{- 1} - τ_{E}^{- 1} τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}})$ is positive for arbitrary $r_{I}$ . For positive $τ_{a}^{- 1} τ_{E}^{- 1} - τ_{E}^{- 1} τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}})$ , we have

b §gt; \frac{τ_{a}^{- 2}}{τ_{a}^{- 1} τ_{E}^{- 1} - τ_{E}^{- 1} τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}})}

Since $τ_{a}$ has to be small, the above condition cannot be satisfied for small $b$ .

Next, we consider the system with large $b$ . Suppose that the firing rate $r_{E}$ and $r_{I}$ in the initial network are of order 1, and $b$ is of order $K$ , where $K$ is a large number. We therefore have $- tr (M_{S F A}) \sim O (1)$ , $A_{11} + A_{22} + A_{33} \sim O (K)$ , and $- det (M_{S F A}) \sim O (K)$ . The discriminant of the characteristic polynomial is

\begin{aligned} (- tr (M_{S F A}))^{2} (A_{11} + A_{22} + A_{33})^{2} - 4 (A_{11} + A_{22} + A_{33})^{3} - 4 (- tr (M_{S F A}))^{3} (- det (M_{S F A})) \\ - 27 (- det (M_{S F A}))^{2} + 18 (- tr (M_{S F A})) (A_{11} + A_{22} + A_{33}) (- det (M_{S F A})) \\ = (A_{11} + A_{22} + A_{33})^{3} [\frac{(- tr (M_{S F A}))^{2}}{A_{11} + A_{22} + A_{33}} - 4 - \frac{4 (- tr (M_{S F A}))^{3} (- det (M_{S F A}))}{(A_{11} + A_{22} + A_{33})^{3}} \\ - \frac{27 (- det (M_{S F A}))^{2}}{(A_{11} + A_{22} + A_{33})^{3}} + \frac{18 (- tr (M_{S F A})) (- det (M_{S F A}))}{(A_{11} + A_{22} + A_{33})^{2}}] \end{aligned}

Clearly, in the large $b$ limit, the discriminant is negative, suggesting that the characteristic polynomial has one real root and two complex conjugate roots (Irving, 2004).

As the input $g_{E}$ increases, the complex conjugate eigenvalues cross the imaginary axis when $tr (M_{S F A}) (A_{11} + A_{22} + A_{33})$ equals $det (M_{S F A})$ . As a result, the system undergoes a supercritical Hopf bifurcation. We numerically confirmed that the resulting limit cycle is stable (Figure 2—figure supplement 1), consistent with previous work (van Vreeswijk and Hansel, 2001). Thus, the system shows oscillatory behavior instead of stable steady state.

Short-term plasticity (STP)

We modeled E-to-E STD following previous work (Tsodyks and Markram, 1997; Varela et al., 1997):

τ_{E} \frac{d r_{E}}{d t} = - r_{E} + {[x J_{E E} r_{E} - J_{E I} r_{I} + g_{E}]}_{+}^{α_{E}}

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + {[J_{I E} r_{E} - J_{I I} r_{I} + g_{I}]}_{+}^{α_{I}}

\frac{d x}{d t} = \frac{1 - x}{τ_{x}} - U_{d} x r_{E}

where $x$ is the depression variable, which is limited to the interval $(0, 1)$ , $τ_{x}$ is the depression time constant, and $U_{d}$ is the depression rate. The steady-state solution $x^{*}$ is given by

x^{*} = \frac{1}{1 + U_{d} r_{E} τ_{x}}

Similarly, we modeled E-to-I STF as

τ_{E} \frac{d r_{E}}{d t} = - r_{E} + {[J_{E E} r_{E} - J_{E I} r_{I} + g_{E}]}_{+}^{α_{E}}

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + {[u J_{I E} r_{E} - J_{I I} r_{I} + g_{I}]}_{+}^{α_{I}}

\frac{d u}{d t} = \frac{1 - u}{τ_{u}} + U_{f} (U_{m a x} - u) r_{E}

where $u$ is the facilitation variable constrained to the interval $(1, U_{m a x})$ , $U_{m a x}$ is the maximal facilitation value, $τ_{u}$ is the time constant of STF, and $U_{f}$ is the facilitation rate. The steady-state solution $u^{*}$ is given by

u^{*} = \frac{1 + U_{f} U_{m a x} r_{E} τ_{u}}{1 + U_{f} r_{E} τ_{u}}

Stability conditions for networks with E-to-E STD

Request a detailed protocol

The Jacobian $M_{S T D}$ of the system with E-to-E STD is given by

M_{S T D} = [\begin{matrix} τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) & - τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} & τ_{E}^{- 1} J_{E E} α_{E} r_{E}^{\frac{2 α_{E} - 1}{α_{E}}} \\ τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} & - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) & 0 \\ - U_{d} x & 0 & - τ_{x}^{- 1} - U_{d} r_{E} \end{matrix}]

and the characteristic polynomial can be written as follows:

λ^{3} - tr (M_{S T D}) λ^{2} + (A_{11} + A_{22} + A_{33}) λ - det (M_{S T D}) = 0

where $tr (M_{S T D})$ and $det (M_{S T D})$ are the trace and the determinant of the Jacobian matrix $M_{S T D}$ , A₁₁, A₂₂, and A₃₃ are the matrix cofactors. More specifically,

tr (M_{S T D}) = τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) - τ_{x}^{- 1} - U_{d} r_{E}

In the case of unstable dynamics, $r_{E}$ goes to infinity due to run-away excitation. However, the depression variable $x$ approaches zero in this limit, as ${lim}_{r_{E} \to \infty} x = {lim}_{r_{E} \to \infty} \frac{1}{1 + U_{d} r_{E} τ_{x}} = 0$ . Therefore, in the large $r_{E}$ limit, $- tr (M_{S T D})$ is positive.

\begin{aligned} A_{11} + A_{22} + A_{33} = & τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) (τ_{x}^{- 1} + U_{d} r_{E}) \\ + τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) (- τ_{x}^{- 1} - U_{d} r_{E}) - τ_{E}^{- 1} J_{E E} α_{E} r_{E}^{\frac{2 α_{E} - 1}{α_{E}}} (- U_{d} x) \\ - τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) + τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} \end{aligned}

Similarly, in the large $r_{E}$ limit, $A_{11} + A_{22} + A_{33}$ is positive.

\begin{aligned} det (M_{S T D}) = & τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) (τ_{x}^{- 1} + U_{d} r_{E}) \\ - τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} τ_{I}^{- 1} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} (τ_{x}^{- 1} + U_{d} r_{E}) - τ_{E}^{- 1} J_{E E} α_{E} r_{E}^{\frac{2 α_{E} - 1}{α_{E}}} U_{d} x τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) \end{aligned}

Similarly, in the large $r_{E}$ limit, $- det (M_{S T D})$ is positive.

According to the Descartes’ rule of signs, the number of positive roots is at most the number of sign changes in the sequences of polynomial’s coefficients. Therefore, there are no positive roots for the above characteristic polynomial and the network dynamics can be stabilized by E-to-E STD.

Characteristic function approximation for networks with E-to-E STD

Request a detailed protocol

As demonstrated above, E-to-E STD is able to restabilize the system, there exists a stable steady state for which the STD variable $x$ is constant $x = x^{*}$ . Because $x$ changes slowly compared to the neuronal dynamics, we can approximate it as constant which results in a natural reduction to a 2D system in which the weights with STD are modified. The stability of this 2D system can be readily characterized by the characteristic function $F (z)$ (Kraynyukova and Tchumatchenko, 2018), which depends on the previous steady state value of $x$ . The characteristic function approximation with E-to-E STD can therefore be written as follows:

F (z) = x J_{E E} [z]_{+}^{α_{E}} - J_{E I} [det (J_{S T D}) \cdot J_{E I}^{- 1} [z]_{+}^{α_{E}} + J_{E I}^{- 1} J_{I I} z - J_{E I}^{- 1} J_{I I} g_{E} + g_{I}]_{+}^{α_{I}} - z + g_{E}

where

det (J_{S T D}) = | \begin{array}{cc} x J_{E E} & - J_{E I} \\ J_{I E} & - J_{I I} \end{array} | = - x J_{E E} J_{I I} + J_{I E} J_{E I}

Note that $det (J_{S T D})$ can now change its sign due to E-to-E STD, the characteristic function can therefore change its bending shape. We used this relation to visualize how E-to-E STD effectively changes the network stability of the reduced system in Figure 2D.

Conditions for ISN in networks with E-to-E STD

Request a detailed protocol

Here, we identify the condition of being in the ISN regime in supralinear networks with E-to-E STD. When the level of inhibition is frozen, the Jacobian of the system reduces to the following:

M_{1} = [\begin{matrix} τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) & τ_{E}^{- 1} J_{E E} α_{E} r_{E}^{\frac{2 α_{E} - 1}{α_{E}}} \\ - U_{d} x & - τ_{x}^{- 1} - U_{d} r_{E} \end{matrix}]

For the system with frozen inhibition, the dynamics are stable if

tr (M_{1}) = τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) - τ_{x}^{- 1} - U_{d} r_{E} < 0

and

det (M_{1}) = τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) (- τ_{x}^{- 1} - U_{d} r_{E}) + τ_{E}^{- 1} J_{E E} α_{E} r_{E}^{\frac{2 α_{E} - 1}{α_{E}}} U_{d} x > 0

Therefore, if the network is an ISN at the fixed point, the following condition has to be satisfied:

x §gt; min (\sqrt{\frac{1}{J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}}}}, \frac{τ_{x} + τ_{E} + τ_{E} τ_{x} U_{d} r_{E}}{τ_{x} J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}}})

Furthermore, we define the largest real part of the eigenvalues of $M_{1}$ as the ISN index for networks with E-to-E STD. More specifically,

ISN index = Re [\frac{τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) - τ_{x}^{- 1} - U_{d} r_{E}}{2} + \sqrt{\frac{1}{4} (τ_{E}^{- 1} (x J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) + τ_{x}^{- 1} + U_{d} r_{E})^{2} - τ_{E}^{- 1} J_{E E} α_{E} r_{E}^{\frac{2 α_{E} - 1}{α_{E}}} U_{d} x}]

Conditions for paradoxical response in networks with E-to-E STD

Request a detailed protocol

Next, we identify the condition of having the paradoxical effect in supralinear networks with E-to-E STD. To that end, we exploit a separation of timescales between the fast neural activity and the slow STP variable. Therefore, set the depression variable to its value at the fixed point corresponding to the fixed point value of $r_{E}$ . The excitatory nullcline is defined as follows

τ_{E} \frac{d r_{E}}{d t} = - r_{E} + {[\frac{1}{1 + τ_{x} U_{d} r_{E}} J_{E E} r_{E} - J_{E I} r_{I} + g_{E}]}_{+}^{α_{E}} = 0

For $r_{E, I} §gt; 0$ , we have

r_{I} = \frac{\frac{1}{1 + τ_{x} U_{d} r_{E}} J_{E E} r_{E} - r_{E}^{\frac{1}{α_{E}}} + g_{E}}{J_{E I}}

The slope of the excitatory nullcline in the $r_{E} / r_{I}$ plane where $x$ axis is $r_{E}$ and $y$ axis is $r_{I}$ can be written as follows

k_{S T D}^{E} = \frac{1}{J_{E I}} (- \frac{J_{E E}}{{(1 + τ_{x} U_{d} r_{E})}^{2}} τ_{x} U_{d} r_{E} + \frac{J_{E E}}{1 + τ_{x} U_{d} r_{E}} - \frac{1}{α_{E}} r_{E}^{\frac{1}{α_{E}} - 1})

Note that the slope of the excitatory nullcline is nonlinear. To have paradoxical effect, the slope of the excitatory nullcline at the fixed point of the system has to be positive. Therefore, the STD variable $x$ at the fixed point has to satisfy the following condition

x §gt; \sqrt{\frac{1}{J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}}}}

The inhibitory nullcline can be written as follows

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + {[J_{I E} r_{E} - J_{I I} r_{I} + g_{I}]}_{+}^{α_{I}} = 0

In the region of rates $r_{E, I} §gt; 0$ , we have

r_{I} = \frac{J_{I E} r_{E} - r_{I}^{\frac{1}{α_{I}}} + g_{I}}{J_{I I}}

The slope of the inhibitory nullcline can be written as follows

k_{S T D}^{I} = \frac{J_{I E}}{J_{I I} + \frac{1}{α_{I}} r_{I}^{\frac{1 - α_{I}}{α_{I}}}}

In addition to the positive slope of the excitatory nullcline, the slope of the inhibitory nullcline at the fixed point of the system has to be larger than the slope of the excitatory nullcline. We therefore have

J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} (τ_{x}^{- 1} + U_{d} r_{E}) > (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) (- \frac{J_{E E} U_{d} r_{E}}{1 + τ_{x} U_{d} r_{E}} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} + \frac{J_{E E}}{1 + τ_{x} U_{d} r_{E}} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} (τ_{x}^{- 1} + U_{d} r_{E}) - (τ_{x}^{- 1} + U_{d} r_{E}))

The above condition is the same as the stability condition of the determinant of the Jacobian of the system with E-to-E STD (Eq. (49)). Therefore, the condition is always satisfied when the system with E-to-E STD is stable.

Based on the condition of being ISN shown in Eq. (55) and the condition of having paradoxical effect shown in Eq. (60), we therefore can conclude that in supralinear networks with E-to-E STD, the paradoxical effect implies inhibitory stabilization, whereas inhibitory stabilization does not necessarily imply paradoxical responses. This is consistent with recent work by Sanzeni et al., 2020, in which threshold-linear networks with STP have been studied. Here, we showed analytically that the conclusion holds for any rectified power-law activation function with positive $α$ .

To visualize the conditions in a two-dimensional plane, we reduced the conditions into a function of $J_{E E}$ and $x$ . For Figure 2G, $r_{E} = 1$ . In Figure 2—figure supplement 5 and Figure 2—figure supplement 8, the depression variable thresholds above which the network exhibits the paradoxical effect were calculated based on Eq. (60).

Uni-stability conditions

Request a detailed protocol

The system is said to be ‘uni-stable’, when it has a single stable fixed point. We first identified the uni-stability condition for networks with global inhibition. To that end, we considered a general network with $N$ excitatory populations and $N$ inhibitory populations. To treat this problem analytically, we did not take STP into account in our analysis. The Jacobian matrix of networks with global inhibition $Q$ , can be written as follows,

Q = [\begin{matrix} J_{E \leftarrow E} & J_{E \leftarrow I} \\ J_{I \leftarrow E} & J_{I \leftarrow I} \end{matrix}]

where $J_{E \leftarrow E}$ , $J_{E \leftarrow I}$ , $J_{I \leftarrow E}$ , and $J_{I \leftarrow I}$ are $N$ by $N$ block matrices defined below.

J_{E \leftarrow E} = [\begin{matrix} a - e & k a & \dots & k a \\ k a & a - e & \dots & k a \\ ⋮ & ⋮ & ⋱ & ⋮ \\ k a & k a & \dots & a - e \end{matrix}]

J_{E \leftarrow I} = - b J_{N, N}

J_{I \leftarrow E} = c J_{N, N}

J_{I \leftarrow I} = [\begin{matrix} - d - f & - d & \dots & - d \\ - d & - d - f & \dots & - d \\ ⋮ & ⋮ & ⋱ & ⋮ \\ - d & - d & \dots & - d - f \end{matrix}]

where $a = τ_{E}^{- 1} J_{E E} α_{E} {[z_{E}]}_{+}^{α_{E} - 1}$ , $b = τ_{E}^{- 1} J_{E I} α_{E} {[z_{E}]}_{+}^{α_{E} - 1}$ , $c = τ_{I}^{- 1} J_{I E} α_{I} {[z_{I}]}_{+}^{α_{I} - 1}$ , $d = τ_{I}^{- 1} J_{I I} α_{I} {[z_{I}]}_{+}^{α_{I} - 1}$ , $e = τ_{E}^{- 1}$ , and $f = τ_{I}^{- 1}$ . Here, $z_{E}$ and $z_{I}$ denote the total current into the excitatory and inhibitory population, respectively. Note that all these parameters are non-negative. Parameter $k$ controls the excitatory connection strength across different populations. $J_{N, N}$ is a $N$ by $N$ matrix of ones.

The eigenvalues of the Jacobian $Q$ are roots of its characteristic polynomial,

det ((J_{E \leftarrow E} - λ 1) (J_{I \leftarrow I} - λ 1) - J_{E \leftarrow I} J_{I \leftarrow E}) = 0

where $1$ represents the identity matrix of size $N$ . The characteristic polynomial can be expanded to:

{[(a - e - k a - λ) (- f - λ)]}^{N - 1} [(a - e + (N - 1) k a - λ) (- N d - f - λ) + N^{2} b c] = 0

We therefore had four distinct eigenvalues:

λ_{1} = a - e - k a

λ_{2} = - f

and

λ_{3 / 4} = \frac{1}{2} [(a - e - f - N d + (N - 1) k a) \pm \sqrt{(a - e - f - N d + (N - 1) k a)^{2} - 4 ((- a f + e f + k a f) - N (a - e) d - N k a f - N (N - 1) k a d + N^{2} b c)}]

Note that the eigenvalues $λ_{1}$ and $λ_{2}$ have an algebraic and geometric multiplicity of ( $N$ –1), whereas the eigenvalues $λ_{3}$ and $λ_{4}$ have an algebraic and geometric multiplicity of 1.

In analogy to networks with global inhibition, the Jacobian matrix of networks with co-tuned inhibition $R$ , can be written as

R = [\begin{matrix} J_{E \leftarrow E} & J_{E \leftarrow I} \\ J_{I \leftarrow E} & J_{I \leftarrow I} \end{matrix}]

where $J_{E \leftarrow E}$ , $J_{E \leftarrow I}$ , $J_{I \leftarrow E}$ , and $J_{I \leftarrow I}$ are $N$ by $N$ block matrices defined as follows:

J_{E \leftarrow E} = [\begin{matrix} a - e & k a & \dots & k a \\ k a & a - e & \dots & k a \\ ⋮ & ⋮ & ⋱ & ⋮ \\ k a & k a & \dots & a - e \end{matrix}]

J_{E \leftarrow I} = [\begin{matrix} - N b + (N - 1) m b & - m b & \dots & - m b \\ - m b & - N b + (N - 1) m b & \dots & - m b \\ ⋮ & ⋮ & ⋱ & ⋮ \\ - m b & - m b & \dots & - N b + (N - 1) m b \end{matrix}]

J_{I \leftarrow E} = [\begin{matrix} N c - (N - 1) m c & m c & \dots & m c \\ m c & N c - (N - 1) m c & \dots & m c \\ ⋮ & ⋮ & ⋱ & ⋮ \\ m c & m c & \dots & N c - (N - 1) m c \end{matrix}]

J_{I \leftarrow I} = [\begin{matrix} - N d + (N - 1) m d - f & - m d & \dots & - m d \\ - m d & - N d + (N - 1) m d - f & \dots & - m d \\ ⋮ & ⋮ & ⋱ & ⋮ \\ - m d & - m d & \dots & - N d + (N - 1) m d - f \end{matrix}]

where $m$ controls the degree of co-tuning in the network. If $m = 0$ , the network decouples into $N$ independent ensembles and inhibition is perfectly co-tuned with excitation. In the case $m = 1$ , inhibition is global and the block matrices become identical to the above case of global inhibition.

The eigenvalues of the matrix $R$ are given as the roots of the characteristic polynomial defined by:

det ((J_{E \leftarrow E} - λ 1) (J_{I \leftarrow I} - λ 1) - J_{E \leftarrow I} J_{I \leftarrow E}) = 0

which yields the following expression:

{[λ^{2} - (a - e - k a - N d + N m d - f) λ - (a - e - k a) (N d - N m d - f) + N^{2} b c {(1 - m)}^{2}]}^{N - 1} [(a - e + (N - 1) k a - λ) (- N d - f - λ) + N^{2} b c] = 0

We therefore had four distinct eigenvalues:

λ_{1 / 2}^{^{'}} = \frac{1}{2} [(a - e - k a - N d + N m d - f) \pm \sqrt{{(a - e - k a + N d - N m d + f)}^{2} - 4 N^{2} b c {(1 - m)}^{2}}]

λ_{3 / 4}^{^{'}} = \frac{1}{2} [(a - e - f - N d + (N - 1) k a) \pm \sqrt{(a - e - f - N d + (N - 1) k a)^{2} - 4 ((- a f + e f + k a f) - N (a - e) d - N k a f - N (N - 1) k a d + N^{2} b c)}]

The eigenvalues $λ_{1}^{^{'}}$ and $λ_{2}^{^{'}}$ have an algebraic and geometric multiplicity of ( $N$ –1), whereas the eigenvalues $λ_{3}^{^{'}}$ and $λ_{4}^{^{'}}$ have an algebraic and geometric multiplicity of 1. We noted that $λ_{3} = λ_{3}^{^{'}}$ , $λ_{4} = λ_{4}^{^{'}}$ .

To compare under which conditions networks with different structures are uni-stable, we examined the different eigenvalues derived above. As $λ_{2} §lt; 0$ , and $λ_{1}^{^{'}} §gt; λ_{2}^{^{'}}$ , we only had to compare $λ_{1}^{^{'}}$ to $λ_{1}$ . For networks with co-tuned inhibition, we have $m §lt; 1$ ,

\begin{aligned} λ_{1}^{^{'}} = & \frac{1}{2} [(a - e - k a - N d + N m d - f) + \sqrt{(a - e - k a + N d - N m d + f)^{2} - 4 N^{2} b c (1 - m)^{2}}] \\ < & \frac{1}{2} [(a - e - k a - N d + N m d - f) + \sqrt{(a - e - k a + N d - N m d + f)^{2}}] = a - e - k a = λ_{1} \end{aligned}

The inequality, $λ_{1}^{^{'}} §lt; λ_{1}$ , indicates that networks with co-tuned inhibition have a broad parameter regime in which they are uni-stable than networks with global inhibition. Note that in the absence of a saturating nonlinearity of the input-output function and in the absence of any additional stabilization mechanisms, systems with positive eigenvalues of the Jacobian are unstable. In this case, networks with co-tuned inhibition have a broad parameter regime of being stable than networks with global inhibition.

To visualize the conditions in a two-dimensional plane, we reduced the conditions into a function of $a$ and $d$ . For Figure 3C, $k = 0.1$ , $m = 0.5$ and $b c = 0.9 a d$ .

Distance to the decision boundary

Request a detailed protocol

To calculate the distance to the decision boundary in Figures 4 and 5, Figure 4—figure supplement 2 and Figure 5—figure supplement 2, we first projected the excitatory activity in Phase two onto a two-dimensional Cartesian coordinate system in which the horizontal axis is the activity of the first excitatory ensemble $r_{E 1}$ and the vertical axis is the activity of the second excitatory ensemble $r_{E 2}$ . We denote the location of the projected data point in the Cartesian coordinate system by ( $x$ , $y$ ), where $x$ and $y$ equal $r_{E 1}$ and $r_{E 2}$ , respectively. The distance $L$ between the projected data and the decision boundary which corresponds to the diagonal line in the coordinate system can be expressed as follows:

L = \sqrt{x^{2} + y^{2}} sin (| 45^{o} - arcsin (\frac{x}{\sqrt{x^{2} + y^{2}}}) |)

Note that the inverse trigonometric function arcsin gives the value of the angle in degrees.

Inhibitory feedback pathways for suppressing unwanted neural activation

Request a detailed protocol

To identify the important neural pathways for the suppression of unwanted neural activation, we analyzed how the activity of the second excitatory ensemble $r_{E 2}$ changes with the input to the first excitatory ensemble $g_{E 1}$ . To that end, we considered a general weight matrix for networks with two interacting ensembles

J = [\begin{matrix} J_{E 1 E 1} & J_{E 1 E 2} & - J_{E 1 I 1} & - J_{E 1 I 2} \\ J_{E 2 E 1} & J_{E 2 E 2} & - J_{E 2 I 1} & - J_{E 2 I 2} \\ J_{I 1 E 1} & J_{I 1 E 2} & - J_{I 1 I 1} & - J_{I 1 I 2} \\ J_{I 2 E 1} & J_{I 2 E 2} & - J_{I 2 I 1} & - J_{I 2 I 2} \end{matrix}]

We can write the change in firing rate of the excitatory population in the second ensemble $δ r_{E 2}$ as a function of the change in the input to the other $δ g_{E 1}$ :

\begin{aligned} δ r_{E 2} = & \frac{1}{det (1 - F J)} [(- f_{E 2}^{^{'}} J_{E 2 E 1}) f_{I 1}^{^{'}} J_{I 1 I 2} f_{I 2}^{^{'}} J_{I 2 I 1} + f_{E 2}^{^{'}} J_{E 2 I 1} (- f_{I 1}^{^{'}} J_{I 1 E 1}) (1 + f_{I 2}^{^{'}} J_{I 2 I 2}) + f_{E 2}^{^{'}} J_{E 2 I 2} (1 + f_{I 1}^{^{'}} J_{I 1 I 1}) (- f_{I 2}^{^{'}} J_{I 2 E 1}) \\ - (- f_{E 2}^{^{'}} J_{E 2 E 1}) (1 + f_{I 1}^{^{'}} J_{I 1 I 1}) (1 + f_{I 2}^{^{'}} J_{I 2 I 2}) - f_{E 2}^{^{'}} J_{E 2 I 1} f_{I 1}^{^{'}} J_{I 1 I 2} (- f_{I 2}^{^{'}} J_{I 2 E 1}) - f_{E 2}^{^{'}} J_{E 2 I 2} (- f_{I 1}^{^{'}} J_{I 1 E 1}) f_{I 2}^{^{'}} J_{I 2 I 1}] f_{E 1}^{^{'}} δ g_{E 1} \end{aligned}

where $1$ is the identity matrix. And $F$ is given by

F = [\begin{matrix} f_{E 1}^{^{'}} & 0 & 0 & 0 \\ 0 & f_{E 2}^{^{'}} & 0 & 0 \\ 0 & 0 & f_{I 1}^{^{'}} & 0 \\ 0 & 0 & 0 & f_{I 2}^{^{'}} \end{matrix}]

where $f_{E 1}^{^{'}}$ , $f_{E 2}^{^{'}}$ , $f_{I 1}^{^{'}}$ and $f_{I 2}^{^{'}}$ are the derivatives of the input-output functions evaluated at the fixed point.

Assuming that $J_{E 1 E 1} = J_{E 2 E 2} = J_{E E}$ , $J_{I 1 E 1} = J_{I 2 E 2} = J_{I E}$ , $J_{E 1 I 1} = J_{E 2 I 2} = J_{E I}$ , $J_{I 1 I 1} = J_{I 2 I 2} = J_{I I}$ , $J_{E 1 E 2} = J_{E 2 E 1} = J_{E E}^{^{'}}$ , $J_{I 1 E 2} = J_{I 2 E 1} = J_{I E}^{^{'}}$ , $J_{E 1 I 2} = J_{E 2 I 1} = J_{E I}^{^{'}}$ and $J_{I 1 I 2} = J_{I 2 I 1} = J_{I I}^{^{'}}$ , we find

\begin{aligned} δ r_{E 2} = & \frac{1}{det (1 - F J)} [(- f_{E 2}^{^{'}} J_{E E}^{^{'}}) f_{I 1}^{^{'}} J_{I I}^{^{'}} f_{I 2}^{^{'}} J_{I I}^{^{'}} + f_{E 2}^{^{'}} J_{E I}^{^{'}} (- f_{I 1}^{^{'}} J_{I E}) (1 + f_{I 2}^{^{'}} J_{I I}) + f_{E 2}^{^{'}} J_{E I} (1 + f_{I 1}^{^{'}} J_{I I}) (- f_{I 2}^{^{'}} J_{I E}^{^{'}}) \\ - (- f_{E 2}^{^{'}} J_{E E}^{^{'}}) (1 + f_{I 1}^{^{'}} J_{I I}) (1 + f_{I 2}^{^{'}} J_{I I}) - f_{E 2}^{^{'}} J_{E I}^{^{'}} f_{I 1}^{^{'}} J_{I I}^{^{'}} (- f_{I 2}^{^{'}} J_{I E}^{^{'}}) - f_{E 2}^{^{'}} J_{E I} (- f_{I 1}^{^{'}} J_{I E}) f_{I 2}^{^{'}} J_{I I}^{^{'}}] f_{E 1}^{^{'}} δ g_{E 1} \end{aligned}

By further assuming that the weight strengths across ensembles are weak and ignoring the corresponding higher-order terms, we get

\begin{aligned} δ r_{E 2} \approx \frac{1}{det (1 - F J)} [f_{E 2}^{^{'}} J_{E I}^{^{'}} (- f_{I 1}^{^{'}} J_{I E}) (1 + f_{I 2}^{^{'}} J_{I I}) + f_{E 2}^{^{'}} J_{E I} (1 + f_{I 1}^{^{'}} J_{I I}) (- f_{I 2}^{^{'}} J_{I E}^{^{'}}) \\ - (- f_{E 2}^{^{'}} J_{E E}^{^{'}}) (1 + f_{I 1}^{^{'}} J_{I I}) (1 + f_{I 2}^{^{'}} J_{I I}) - f_{E 2}^{^{'}} J_{E I} (- f_{I 1}^{^{'}} J_{I E}) f_{I 2}^{^{'}} J_{I I}^{^{'}}] f_{E 1}^{^{'}} δ g_{E 1} \\ = \frac{1}{det (1 - F J)} [(\frac{J_{I I}^{^{'}}}{J_{E I}^{^{'}}} f_{I 2}^{^{'}} - (\frac{1}{J_{E I}} + f_{I 2}^{^{'}} \frac{J_{I I}}{J_{E I}})) J_{E I}^{^{'}} J_{E I} J_{I E} f_{E 2}^{^{'}} f_{I 1}^{^{'}} \\ + (\frac{J_{E E}^{^{'}}}{J_{I E}^{^{'}}} (1 + J_{I I} f_{I 2}^{^{'}}) - J_{E I} f_{I 2}^{^{'}}) J_{I E}^{^{'}} f_{E 2}^{^{'}} (1 + f_{I 1}^{^{'}} J_{I I})] f_{E 1}^{^{'}} δ g_{E 1} \end{aligned}

Note that $\frac{J_{E E}^{^{'}}}{J_{I E}^{^{'}}}$ and $\frac{J_{I I}^{^{'}}}{J_{E I}^{^{'}}}$ are terms regulating the respective excitatory and inhibitory input from one ensemble to the excitatory and inhibitory population in another ensemble. The term $det (1 - F J)$ is positive to ensure the stability of the system.

To suppress the activity of the excitatory population in the second ensemble $r_{E 2}$ , in other words, to ensure that $δ r_{E 2} §lt; 0$ , $J_{I E}^{^{'}}$ or/and $J_{E I}^{^{'}}$ have to be large. Therefore, we identified $J_{I E}^{^{'}}$ and $J_{E I}^{^{'}}$ as important synaptic connections which lead to suppression of the unwanted neural activation, suggesting that inhibition can be provided via $J_{I E}^{^{'}}$ through the $E 1$ - $I 2$ - $E 2$ pathway or via $J_{E I}^{^{'}}$ through the $E 1$ - $I 1$ - $E 2$ pathway.

For Figures 4 and 5, the rate-based model consists of two ensembles, each of which is composed of 100 excitatory and 25 inhibitory neurons with all-to-all connectivity.

Spiking neural network model

Request a detailed protocol

The spiking neural network model was composed of $N_{E}$ excitatory and $N_{I}$ inhibitory leaky integrate-and-fire neurons. Neurons were randomly connected with probability of 20%. The dynamics of membrane potential of neuron i, $U_{i}$ , as defined by Zenke et al., 2015:

τ^{m} \frac{d U_{i}}{d t} = (U^{rest} - U_{i}) + g_{i}^{ext} (t) (U^{exc} - U_{i}) + g_{i}^{inh} (t) (U^{inh} - U_{i})

Here, $τ^{m}$ is the membrane time constant and $U^{rest}$ is the resting potential. Spikes are triggered when the membrane potential reaches the spiking threshold $U^{thr}$ . After a spike is emitted, the membrane potential is reset to $U^{rest}$ and the neuron enters a refractory period of $τ^{ref}$ . Inhibitory neurons obeyed the same integrate-and-fire formalism but with a shorter membrane time constant.

Excitatory synapses contain a fast AMPA component and a slow NMDA component. The dynamics of the excitatory conductance are described by:

τ^{ampa} \frac{d g_{i}^{ampa}}{d t} = - g_{i}^{ampa} + \sum_{j \in exc} J_{i j} S_{j} (t)

τ^{nmda} \frac{d g_{i}^{nmda}}{d t} = - g_{i}^{nmda} + g_{i}^{ampa}

g_{i}^{exc} (t) = ξ g_{i}^{ampa} (t) + (1 - ξ) g_{i}^{nmda} (t)

Here, $J_{i j}$ denotes the synaptic strength from neuron $j$ to neuron i. If the connection does not exist, $J_{i j}$ was set to 0. $S_{j} (t)$ is the spike train of neuron $j$ , which is defined as $S_{j} (t) = \sum_{k} δ (t - t_{j}^{k})$ , where $δ$ is the Dirac delta function and $t_{j}^{k}$ the spikes times $k$ of neuron $j$ . $ξ$ is a weighting parameter. The dynamics of inhibitory conductances are governed by:

τ^{gaba} \frac{d g_{i}^{inh}}{d t} = - g_{i}^{inh} + \sum_{j \in inh} J_{i j} S_{j} (t)

In the spiking neural network models, SFA of excitatory neurons is modeled as follows,

τ^{m} \frac{d U_{i}}{d t} = (U^{rest} - U_{i}) + g_{i}^{ext} (t) (U^{exc} - U_{i}) + (g_{i}^{inh} (t) + a_{i} (t)) (U^{inh} - U_{i})

\frac{d a_{i}}{d t} = - \frac{a_{i}}{τ_{a}} + b S_{i} (t)

where i is the index of excitatory neurons.

The dynamics of E-to-E STD are given by

\frac{d x_{i j}}{d t} = \frac{1 - x_{i j}}{τ_{x}} - U_{d} x_{i j} S_{j} (t)

τ^{ampa} \frac{d g_{i}^{ampa}}{d t} = - g_{i}^{ampa} + \sum_{j \in exc} x_{i j} J_{i j} S_{j} (t)

where i represents the index of excitatory neurons.

The dynamics of E-to-I STF are governed by

\frac{d u_{i j}}{d t} = \frac{1 - u_{i j}}{τ_{u}} + U_{f} (U_{m a x} - u_{i j}) S_{j} (t)

τ^{ampa} \frac{d g_{i}^{ampa}}{d t} = - g_{i}^{ampa} + \sum_{j \in exc} u_{i j} J_{i j} S_{j} (t)

where i denotes the index of inhibitory neurons.

For Figure 6, each excitatory and inhibitory neuron received external excitatory input from 300 neurons firing with Poisson statistics at an average firing rate of 0.1 Hz at baseline. During stimulation, the excitatory neurons corresponding to the activated ensemble received external excitatory input from 300 neurons firing with Poisson statistics at an average firing rate of 0.5 Hz. The ensemble activity is computed from the instantaneous firing rates of the respective ensembles with 10ms bin size. The difference in ensemble activity for the peak amplitude is calculated by subtracting the average maximal ensemble activity of the unstimulated ensembles from the maximal ensemble activity of the activated ensemble. Similarly, the difference in ensemble activity for the fixed point is calculated by subtracting the average ensemble activity of the unstimulated ensembles at the fixed point from the ensemble activity of the activated ensemble at the fixed point. Fixed point activity is computed by averaging the activity of the middle 1 second within the 2-second stimulation period.

For Figure 2—figure supplement 10, each excitatory and inhibitory neuron received external excitatory input from 300 neurons firing with Poisson statistics at an average firing rate of 0.1 Hz at the baseline. During stimulation, each excitatory neuron received external excitatory input from 300 neurons firing with Poisson statistics at an average firing rate of 0.3 Hz.

For Figure 6—figure supplement 1, the firing rates of 300 neurons are varying from $4 / 15$ Hz to $7 / 15$ Hz.

Simulations

Request a detailed protocol

Simulations were performed in Python and Mathematica. All differential equations were implemented by Euler integration with a time step of 0.1 ms. All simulation parameters are listed in Tables 1–5 and Appendix 5—Tables 1–10. The simulation source code to reproduce the figures is publicly available at https://github.com/fmi-basel/gzenke-nonlinear-transient-amplification (Wu, 2021 copy archived at swh:1:rev:6ff6ff10b9f4994a0f948a987a66cc82f98451e1).

Table 1

Parameters for Figure 1C–E.

Symbol	Value	Unit	Description
$J_{E E}$	1.8	-	E-to-E connection strength
$J_{I E}$	1.0	-	E-to-I connection strength
$J_{E I}$	1.0	-	I-to-E connection strength
$J_{I I}$	0.6	-	I-to-I connection strength
$α_{E}$	2	-	Power of excitatory input-output function
$α_{I}$	2	-	Power of inhibitory input-output function
$τ_{E}$	20	ms	Time constant of excitatory firing dynamics
$τ_{I}$	10	ms	Time constant of inhibitory firing dynamics
$g_{E}^{b s}$	1.55	-	Input to the E population at baseline
$g_{E}^{s t i m}$	3.0	-	Input to the E population during stimulation
$g_{I}$	2.0	-	Input to the I population
Parameters for Figure 1F
$J_{I E}$	0.45	-	E-to-I connection strength
$J_{E I}$	1.0	-	I-to-E connection strength
$J_{I I}$	1.5	-	I-to-I connection strength

Table 2

Parameters for Figure 2.

Symbol	Value	Unit	Description
$τ_{a}$	200	ms	Time constant of SFA
$b$	1.0	-	Strength of SFA
$τ_{x}$	200	ms	Time constant of STD
$U_{d}$	1.0	-	Depression rate
$τ_{u}$	200	ms	Time constant of STF
$U_{f}$	1.0	-	Facilitation rate
$U_{m a x}$	6.0	-	Maximal facilitation value
Note that these values are also applied elsewhere unless mentioned otherwise.

Table 3

Parameters for Figure 3 bi/multi-stable example.

Symbol	Value	Unit	Description
$J_{E E}$	1.4	-	Within-ensemble E-to-E connection strength
$J_{I E}$	0.6	-	Within-ensemble E-to-I connection strength
$J_{E I}$	1.0	-	Within-ensemble I-to-E connection strength
$J_{I I}$	0.6	-	Within-ensemble I-to-I connection strength
$J_{E E}^{^{'}}$	0.14	-	Inter-ensemble E-to-E connection strength
$J_{I E}^{^{'}}$	0.6	-	Inter-ensemble E-to-I connection strength
$J_{E I}^{^{'}}$	1.0	-	Inter-ensemble I-to-E connection strength
$J_{I I}^{^{'}}$	0.6	-	Inter-ensemble I-to-I connection strength
$g_{E 1}^{b s}$	2.2	-	Input to the E1 population at baseline
$g_{E 1}^{s t i m}$	3.0	-	Input to the E1 population during stimulation
$g_{E 2}$	2.2	-	Input to the E2 population
$g_{I}$	2.0	-	Input to the I population
Parameters for Figure 3 uni-stable example
$J_{E E}$	1.3	-	Within-ensemble E-to-E connection strength
$J_{E E}^{^{'}}$	0.13	-	Inter-ensemble E-to-E connection strength

Table 4

Parameters for Figures 4 and 5.

Symbol	Value	Unit	Description
$N_{E}$	200	-	Number of excitatory neurons
$N_{I}$	50	-	Number of inhibitory neurons
$N$	2	-	Number of ensembles
$J_{E E}$	$1.2 / (N_{E} / 2 - 1)$	-	Within-ensemble E-to-E connection strength
$J_{I E}$	$1.0 / (N_{E} / 2)$	-	Within-ensemble E-to-I connection strength
$J_{E I}$	$1.0 / (N_{I} / 2)$	-	Within-ensemble I-to-E connection strength
$J_{I I}$	$1.0 / (N_{I} / 2 - 1)$	-	Within-ensemble I-to-I connection strength
$J_{E E}^{^{'}}$	$0.36 / (N_{E} / 2 - 1)$	-	Inter-ensemble E-to-E connection strength
$J_{I E}^{^{'}}$	$0.4 / (N_{E} / 2)$	-	Inter-ensemble E-to-I connection strength
$J_{E I}^{^{'}}$	$0.1 / (N_{I} / 2)$	-	Inter-ensemble I-to-E connection strength
$J_{I I}^{^{'}}$	$0.1 / (N_{I} / 2)$	-	Inter-ensemble I-to-I connection strength
$g_{I}$	2.0	-	Input to the I population
Parameters for Figure 4
$g_{E 1}^{b s}$	1.35	-	Input to the E1 population
$g_{E 1}^{s t i m}$	4.0	-	Input to the E1 population during stimulation
$g_{E 2}$	1.35	-	Input to the E2 population
Parameters for Figure 5
$g_{E 1}^{b s}$	1.35	-	Input to the E1 population at baseline
$g_{E 1}^{s t i m}$	1.35 + (4.0–1.35) (1- $p$ )	-	Input to the E1 population during stimulation
$g_{E 2}^{b s}$	1.35	-	Input to the E2 population at baseline
$g_{E 2}^{s t i m}$	1.35 + (4.0–1.35) $p$	-	Input to the E2 population during stimulation
Here, p is a parameter between 0 and 1 controlling the additional inputs to E1 and E2.

Table 5

Parameters for Figure 6.

Symbol	Value	Unit	Description
$N_{E}$	400	-	Number of excitatory neurons
$N_{I}$	100	-	Number of inhibitory neurons
$U^{rest}$	–70	mV	Resting membrane potential
$U^{exc}$	0	mV	Excitatory reversal potential
$U^{inh}$	–80	mV	Inhibitory reversal potential
$τ^{ref}$	3	ms	Duration of refractory period
$τ_{exc}^{m}$	20	ms	Membrane time constant of excitatory neurons
$τ_{inh}^{m}$	10	ms	Membrane time constant of inhibitory neurons
$τ^{ampa}$	5	ms	Time constant of AMPA receptor
$τ^{gaba}$	10	ms	Time constant of GABA receptor
$τ^{nmda}$	100	ms	Time constant of NMDA receptor
$ξ$	0.5	-	Receptor weighting factor
$J_{E E}$	0.19	-	Within-ensemble E-to-E connection strength
$J_{I E}$	0.10	-	Within-ensemble E-to-I connection strength
$J_{E I}$	0.10	-	Within-ensemble I-to-E connection strength
$J_{I I}$	0.06	-	Within-ensemble I-to-I connection strength
$J_{E E}^{^{'}}$	0.019	-	Inter-ensemble E-to-E connection strength
$J_{I E}^{^{'}}$	0.05	-	Inter-ensemble E-to-I connection strength
$J_{E I}^{^{'}}$	0.04	-	Inter-ensemble I-to-E connection strength
$J_{I I}^{^{'}}$	0.006	-	Inter-ensemble I-to-I connection strength

Appendix 1

Stability conditions in networks with E-to-I STF

The dynamics of supralinear networks with E-to-I STF can be described as follows:

τ_{E} \frac{d r_{E}}{d t} = - r_{E} + [J_{E E} r_{E} - J_{E I} r_{I} + g_{E}]_{+}^{α_{E}}

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + [u J_{I E} r_{E} - J_{I I} r_{I} + g_{I}]_{+}^{α_{I}}

\frac{d u}{d t} = \frac{1 - u}{τ_{u}} + U_{f} (U_{m a x} - u) r_{E}

The Jacobian $M_{S T F}$ of the system with E-to-I STF is given by:

M_{S T F} = [\begin{matrix} τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) & - τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} & 0 \\ τ_{I}^{- 1} u J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} & - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) & τ_{I}^{- 1} J_{I E} r_{E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} \\ U_{f} (U_{m a x} - u) & 0 & - τ_{u}^{- 1} - U_{f} r_{E} \end{matrix}]

The characteristic polynomial for the system with E-to-I STF can be written as follows:

λ^{3} - tr (M_{S T F}) λ^{2} + (A_{11} + A_{22} + A_{33}) λ - det (M_{S T F}) = 0

where $tr (M_{S T F})$ and $det (M_{S T F})$ are the trace and the determinant of the Jacobian matrix $M_{S F A}$ , A₁₁, A₂₂, and A₃₃ are the matrix cofactors. More specifically,

\begin{aligned} tr (M_{S T F}) & = τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) - τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) - τ_{u}^{- 1} - U_{f} r_{E} \\ \propto τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} / r_{I}^{\frac{α_{I} - 1}{α_{I}}} - r_{I}^{\frac{1 - α_{I}}{α_{I}}}) - τ_{I}^{- 1} (r_{I}^{\frac{1 - α_{I}}{α_{I}}} + J_{I I} α_{I}) - τ_{u}^{- 1} r_{I}^{\frac{1 - α_{I}}{α_{I}}} - U_{f} r_{E} r_{I}^{\frac{1 - α_{I}}{α_{I}}} \end{aligned}

Assuming that $α_{E} = α_{I} = α$ , we then have

tr (M_{S T F}) \propto τ_{E}^{- 1} [J_{E E} α (\frac{r_{E}}{r_{I}})^{\frac{α - 1}{α}} - r_{I}^{\frac{1 - α}{α}}] - τ_{I}^{- 1} (r_{I}^{\frac{1 - α}{α}} + J_{I I} α) - τ_{u}^{- 1} r_{I}^{\frac{1 - α}{α}} - U_{f} r_{E} r_{I}^{\frac{1 - α}{α}}

Substituting the firing rates with the current into excitatory population $z$ , we then have

\begin{aligned} tr (M_{S T F}) \propto & τ_{E}^{- 1} [J_{E E} α (\frac{z}{det (J_{S T F}) \cdot J_{E I}^{- 1} [z]_{+}^{α} + J_{E I}^{- 1} J_{I I} z - J_{E I}^{- 1} J_{I I} g_{E} + g_{I}})^{α - 1} - r_{I}^{\frac{1 - α}{α}}] \\ - τ_{I}^{- 1} (r_{I}^{\frac{1 - α}{α}} + J_{I I} α) - τ_{u}^{- 1} r_{I}^{\frac{1 - α}{α}} - U_{f} r_{E} r_{I}^{\frac{1 - α}{α}} \end{aligned}

det (J_{S T F}) = | \begin{array}{cc} J_{E E} & - J_{E I} \\ u J_{I E} & - J_{I I} \end{array} | = - J_{E E} J_{I I} + u J_{I E} J_{E I}

In the large $r_{E}$ limit, $z$ is large, ${lim}_{r_{E} \to \infty} u = {lim}_{r_{E} \to \infty} \frac{1 + U_{f} U_{m a x} r_{E} τ_{u}}{1 + U_{f} r_{E} τ_{u}} \approx U_{m a x}$ . Therefore, we can guarantee that $det (J_{S T F})$ becomes positive for sufficiently large $U_{m a x}$ . Since the denominator $det (J_{S T F}) \cdot J_{E I}^{- 1} [z]_{+}^{α} + J_{E I}^{- 1} J_{I I} z - J_{E I}^{- 1} J_{I I} g_{E} + g_{I}$ grows faster than the numerator for $z ≫ 1$ , $tr (M_{S T F})$ becomes negative for large $r_{E}$ .

\begin{aligned} A_{11} + A_{22} + A_{33} = & τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) (τ_{u}^{- 1} + U_{f} r_{E}) \\ + τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) (- τ_{u}^{- 1} - U_{f} r_{E}) \\ - τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) + τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} τ_{I}^{- 1} u J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} \end{aligned}

Similarly, in the large $r_{E}$ limit, $A_{11} + A_{22} + A_{33}$ is positive.

\begin{aligned} det (M_{S T F}) = & τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) τ_{I}^{- 1} (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) (τ_{u}^{- 1} + U_{f} r_{E}) \\ + τ_{E}^{- 1} J_{E I} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} (τ_{I}^{- 1} u J_{I E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} (- τ_{u}^{- 1} - U_{f} r_{E}) - τ_{I}^{- 1} J_{I E} r_{E} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} U_{f} (U_{m a x} - u)) \end{aligned}

Similarly, in the large $r_{E}$ limit, $det (M_{S T F})$ is negative.

Therefore, similar to E-to-E STD, networks dynamics can also be stabilized by E-to-I STF.

Appendix 2

Conditions for ISN in networks with E-to-I STF

Here, we identify the condition of being ISN in supralinear networks with E-to-I STF. If inhibition is frozen, in other words, if feedback inhibition is absent, the Jacobian of the system becomes as follows:

M_{2} = [\begin{matrix} τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) & 0 \\ U_{f} (U_{m a x} - u) & - τ_{u}^{- 1} - U_{f} r_{E} \end{matrix}]

For the system with frozen inhibition, the dynamics are stable if

tr (M_{2}) = τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) - τ_{u}^{- 1} - U_{f} r_{E} < 0

and

det (M_{2}) = τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) (- τ_{u}^{- 1} - U_{f} r_{E}) > 0

Therefore, if the network is an ISN at the fixed point, the following condition has to be satisfied:

τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) > 0

Note that this condition is independent of the facilitation variable $u$ of E-to-I STF. We further define the ISN index for the system with E-to-I STF as follows:

ISN index = τ_{E}^{- 1} (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1)

Appendix 3

Conditions for paradoxical response in networks with E-to-I STF

Next, we identify the condition of having the paradoxical effect in supralinear networks with E-to-I STF. The excitatory nullcline is defined by

τ_{E} \frac{d r_{E}}{d t} = - r_{E} + {[J_{E E} r_{E} - J_{E I} r_{I} + g_{E}]}_{+}^{α_{E}} = 0

For $r_{E, I} §gt; 0$ , we have

r_{I} = \frac{J_{E E} r_{E} - r_{E}^{\frac{1}{α_{E}}} + g_{E}}{J_{E I}}

The slope of the excitatory nullcline in the $r_{E} / r_{I}$ plane where $x$ axis is $r_{E}$ and $y$ axis is $r_{I}$ can be written as follows

k_{S T F}^{E} = \frac{J_{E E} - \frac{1}{α_{E}} r_{E}^{\frac{1}{α_{E}} - 1}}{J_{E I}}

Note that the slope of the excitatory nullcline is nonlinear. To have paradoxical effect, the slope of the excitatory nullcline at the fixed point of the system has to be positive. We therefore have

J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1 §gt; 0

We exploit a separation of timescales between fast neural activity and slow short-term plasticity variable, we therefore set the facilitation variable to the value at its fixed point corresponding to the dynamical value of $r_{E}$ . Then we can write the inhibitory nullcline as follows

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + {[\frac{1 + U_{f} U_{m a x} r_{E} τ_{u}}{1 + U_{f} r_{E} τ_{u}} J_{I E} r_{E} - J_{I I} r_{I} + g_{I}]}_{+}^{α_{I}} = 0

In the region of rates $r_{E, I} §gt; 0$ , we have

r_{I} = \frac{\frac{1 + U_{f} U_{m a x} r_{E} τ_{u}}{1 + U_{f} r_{E} τ_{u}} J_{I E} r_{E} - r_{I}^{\frac{1}{α_{I}}} + g_{I}}{J_{I I}}

The slope of the inhibitory nullcline can be written as follows

k_{S T F}^{I} = \frac{\frac{1 + U_{f} U_{m a x} r_{E} τ_{u}}{1 + U_{f} r_{E} τ_{u}} J_{I E} + \frac{U_{f} U_{m a x} τ_{u} - U_{f} τ_{u}}{{(1 + U_{f} r_{E} τ_{u})}^{2}} J_{I E} r_{E}}{J_{I I} + \frac{1}{α_{I}} r_{I}^{\frac{1 - α_{I}}{α_{I}}}}

\begin{aligned} - (J_{E E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} - 1) (1 + J_{I I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}}) + J_{I E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} \frac{1 + U_{f} U_{m a x} r_{E} τ_{u}}{1 + U_{f} r_{E} τ_{u}} J_{E I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} \\ + J_{I E} α_{E} r_{E}^{\frac{α_{E} - 1}{α_{E}}} \frac{U_{f} U_{m a x} τ_{u} - U_{f} τ_{u}}{(1 + U_{f} r_{E} τ_{u})^{2}} J_{E I} α_{I} r_{I}^{\frac{α_{I} - 1}{α_{I}}} r_{E} > 0 \end{aligned}

The above condition is the same as the stability condition of the determinant of the Jacobian of the system with E-to-I STF (Eq. (112)). Therefore, the condition is always satisfied when the system with E-to-I STF is stable.

Note that the condition of being ISN shown in Eq. (116) is identical to the condition of having paradoxical effect shown in Eq. (121). Therefore, in networks with E-to-I STF alone, paradoxical effect implies ISN and ISN implies paradoxical effect. We thus use paradoxical effect as a proxy for inhibitory stabilization.

Appendix 4

Change in steady-state activity of unstimulated co-tuned neurons

To analyze the pattern completion in supralinear networks, we considered a network with one excitatory population and one inhibitory population. Neurons in the excitatory population are co-tuned to the same stimulus feature and are separated into two subsets denoting by E₁₁ and E₁₂. The dynamics of the system can be described as follows:

τ_{E} \frac{d r_{E_{11}}}{d t} = - r_{E_{11}} + {[J_{E_{11} E_{11}} r_{E_{11}} + J_{E_{11} E_{12}} r_{E_{12}} - J_{E_{11} I} r_{I} + g_{E_{11}}]}_{+}^{α_{E}}

τ_{E} \frac{d r_{E_{12}}}{d t} = - r_{E_{12}} + [J_{E_{12} E_{11}} r_{E_{11}} + J_{E_{12} E_{12}} r_{E_{12}} - J_{E_{12} I} r_{I} + g_{E_{12}}]_{+}^{α_{E}}

τ_{I} \frac{d r_{I}}{d t} = - r_{I} + [J_{I E_{11}} r_{E_{11}} + J_{I E_{12}} r_{E_{12}} - J_{I I} r_{I} + g_{I}]_{+}^{α_{I}}

The change in the firing rate of the Subset 2 in the excitatory population $δ r_{E_{12}}$ can be written as a function of the change in the input to the Subset 1 $δ g_{E_{11}}$ :

\begin{aligned} δ r_{E_{12}} & = \frac{1}{det (1 - F J)} [- f_{E_{12}}^{^{'}} J_{E_{12} I} f_{I}^{^{'}} J_{I E_{11}} - (- f_{E_{12}}^{^{'}} J_{E_{12} E_{11}}) (1 + f_{I}^{^{'}} J_{I I})] f_{E_{11}}^{^{'}} δ g_{E_{11}} \\ = \frac{1}{det (1 - F J)} [J_{E_{12} E_{11}} + J_{E_{12} E_{11}} J_{I I} f_{I}^{^{'}} - J_{E_{12} I} J_{I E_{11}} f_{I}^{^{'}}] f_{E_{11}}^{^{'}} f_{E_{12}}^{^{'}} δ g_{E_{11}} \end{aligned}

where $1$ is the identity matrix. And $F$ is given by

F = [\begin{matrix} f_{E_{11}}^{^{'}} & 0 & 0 \\ 0 & f_{E_{12}}^{^{'}} & 0 \\ 0 & 0 & f_{I}^{^{'}} \end{matrix}]

where $f_{E_{11}}^{^{'}}$ , $f_{E_{12}}^{^{'}}$ , and $f_{I}^{^{'}}$ are the derivatives of the input-output functions evaluated at the fixed point. The term $det (1 - F J)$ is positive to ensure the stability of the system.

Clearly, if the term $J_{E_{12} E_{11}} + J_{E_{12} E_{11}} J_{I I} f_{I}^{^{'}} - J_{E_{12} I} J_{I E_{11}} f_{I}^{^{'}}$ is positive (negative), increasing the input to the Subset 1 leads to an increase (a decrease) in the activity of neurons in the Subset 2. As the input to the Subset 1 increases, the firing rate of the inhibitory population $r_{I}$ and also $f_{I}^{^{'}}$ will increase. In the presence of E-to-E STD or E-to-I STF, $J_{E_{12} E_{11}}$ or $J_{I E_{11}}$ will decrease or increase with the input to the Subset 1. As a result, the sign of $J_{E_{12} E_{11}} + J_{E_{12} E_{11}} J_{I I} f_{I}^{^{'}} - J_{E_{12} I} J_{I E_{11}} f_{I}^{^{'}}$ can switch from positive to negative as the input to the Subset 1 increases, indicating that the effect on the activity of the co-tuned unstimulated neurons in the same ensemble can switch from potentiation to suppression. Note that this behavior is different from linear networks in which the change is independent of the input or firing rates.

Appendix 5

Appendix 5—table 1

Parameters for Figure 1—figure supplement 1.

Symbol	Value	Unit	Description
$J_{E E}$	0.5	-	E-to-E connection strength
$J_{I E}$	0.45	-	E-to-I connection strength
$J_{E I}$	1.0	-	I-to-E connection strength
$J_{I I}$	1.5	-	I-to-I connection strength
$g_{E}^{b s}$	0.5	-	Input to the E population at baseline
$g_{I}^{b s}$	1.5	-	Input to the I population at baseline

Appendix 5—table 2

Parameters for Figure 2—figure supplement 2.

Symbol	Value	Unit	Description
$g_{E}^{s t i m}$	2.0	-	Input to the E population during stimulation
Note that values of the unlisted parameters are the same as Tables 1–2.

Appendix 5—table 3

Parameters for Figure 2—figure supplement 3 SSN example.

Symbol	Value	Unit	Description
$J_{E E}$	1.8	-	E-to-E connection strength
$J_{I E}$	2.0	-	E-to-I connection strength
$J_{E I}$	1.0	-	I-to-E connection strength
$J_{I I}$	1.0	-	I-to-I connection strength

Appendix 5—table 4

Parameters for Figure 2—figure supplement 5.

Symbol	Value	Unit	Description
$g_{E}^{b s}$	1.8	-	Input to the E population at baseline
Note that values of the unlisted parameters are the same as Tables 1–2.

Appendix 5—table 5

Parameters for Figure 2—figure supplement 10.

Symbol	Value	Unit	Description
$N_{E}$	400	-	Number of excitatory neurons
$N_{I}$	100	-	Number of inhibitory neurons
$J_{E E}$	0.05	-	E-to-E connection strength
$J_{I E}$	0.02	-	E-to-I connection strength
$J_{E I}$	0.05	-	I-to-E connection strength
$J_{I I}$	0.03	-	I-to-I connection strength

Appendix 5—table 6

Parameters for Figure 2—figure supplement 11.

Symbol	Value	Unit	Description
$J_{E E}$	0.9	-	E-to-E connection strength
$J_{I E}$	1.2	-	E-to-I connection strength
$J_{E I}$	0.5	-	I-to-E connection strength
$J_{I I}$	0.5	-	I-to-I connection strength
$τ_{E}$	20	ms	Time constant of excitatory firing dynamics
$τ_{I}$	60	ms	Time constant of inhibitory firing dynamics
$g_{E}^{b s}$	1.0	-	Input to the E population at baseline
$g_{E}^{s t i m}$	2.0	-	Input to the E population during stimulation
$g_{I}$	2.0	-	Input to the I population
Note that values of the unlisted parameters are the same as Tables 1–2.

Appendix 5—table 7

Parameters for Figure 2—figure supplement 12.

Symbol	Value	Unit	Description
$J_{E E}$	1.0	-	E-to-E connection strength
$J_{I E}$	1.2	-	E-to-I connection strength
$J_{E I}$	0.5	-	I-to-E connection strength
$J_{I I}$	1.0	-	I-to-I connection strength
$g_{E}^{b s}$	0.5	-	Input to the E population at baseline
$g_{I}^{b s}$	1.0	-	Input to the I population at baseline

Appendix 5—table 8

Parameters for Figure 3—figure supplement 1 global inhibition example.

Symbol	Value	Unit	Description
$J_{E E}$	1.6	-	Within-ensemble E-to-E connection strength
$J_{I E}$	1.0	-	Within-ensemble E-to-I connection strength
$J_{E I}$	1.0	-	Within-ensemble I-to-E connection strength
$J_{I I}$	1.2	-	Within-ensemble I-to-I connection strength
$J_{E E}^{^{'}}$	0.16	-	Inter-ensemble E-to-E connection strength
$J_{I E}^{^{'}}$	1.0	-	Inter-ensemble E-to-I connection strength
$J_{E I}^{^{'}}$	1.0	-	Inter-ensemble I-to-E connection strength
$J_{I I}^{^{'}}$	1.2	-	Inter-ensemble I-to-I connection strength
$g_{E 1}^{b s}$	1.5	-	Input to the E1 population at baseline
$g_{E 2}$	1.5	-	Input to the E2 population
$g_{I 1}$	2.5	-	Input to the I1 population
$g_{I 2}$	2.5	-	Input to the I2 population
Parameters for Figure 3—figure supplement 1 co-tuned example
$J_{I E}$	1.0 * (4/3)	-	Within-ensemble E-to-I connection strength
$J_{E I}$	1.0 * (4/3)	-	Within-ensemble I-to-E connection strength
$J_{I I}$	1.2 * (4/3)	-	Within-ensemble I-to-I connection strength
$J_{I E}^{^{'}}$	1.0 * (2/3)	-	Inter-ensemble E-to-I connection strength
$J_{E I}^{^{'}}$	1.0 * (2/3)	-	Inter-ensemble I-to-E connection strength
$J_{I I}^{^{'}}$	1.2 * (2/3)	-	Inter-ensemble I-to-I connection strength

Appendix 5—table 9

Parameters for Figure 4—figure supplement 1.

Symbol	Value	Unit	Description
$J_{E E}$	$1.5 / (N_{E} / 2 - 1)$	-	Within-ensemble E-to-E connection strength
$J_{I E}$	$1.0 / (N_{E} / 2)$	-	Within-ensemble E-to-I connection strength
$J_{E I}$	$1.0 / (N_{I} / 2)$	-	Within-ensemble I-to-E connection strength
$J_{I I}$	$1.0 / (N_{I} / 2 - 1)$	-	Within-ensemble I-to-I connection strength
$J_{E E}^{^{'}}$	$0.1 / (N_{E} / 2 - 1)$	-	Inter-ensemble E-to-E connection strength
$J_{I E}^{^{'}}$	$0.3 / (N_{E} / 2)$	-	Inter-ensemble E-to-I connection strength
$J_{E I}^{^{'}}$	$0.3 / (N_{I} / 2)$	-	Inter-ensemble I-to-E connection strength
$J_{I I}^{^{'}}$	$0.1 / (N_{I} / 2)$	-	Inter-ensemble I-to-I connection strength
$g_{E 1}^{b s}$	1.5	-	Input to the E1 population at baseline
$g_{E 2}$	1.5	-	Input to the E2 population
$g_{I}$	2.0	-	Input to the I population

Appendix 5—table 10

Parameters for Figure 6—figure supplement 1.

Symbol	Value	Unit	Description
$J_{E E}$	0.20	-	Within-ensemble E-to-E connection strength
$J_{I E}$	0.09	-	Within-ensemble E-to-I connection strength
$J_{E I}$	0.10	-	Within-ensemble I-to-E connection strength
$J_{I I}$	0.10	-	Within-ensemble I-to-I connection strength
$J_{E E}^{^{'}}$	0.02	-	Inter-ensemble E-to-E connection strength
$J_{I E}^{^{'}}$	0.054	-	Inter-ensemble E-to-I connection strength
$J_{E I}^{^{'}}$	0.07	-	Inter-ensemble I-to-E connection strength
$J_{I I}^{^{'}}$	0.01	-	Inter-ensemble I-to-I connection strength
Note that values of the unlisted parameters are the same as Table 5.

Data availability

This project is a theory project without data. All simulation code has been deposited on GitHub under https://github.com/fmi-basel/gzenke-nonlinear-transient-amplification, (copy archived at swh:1:rev:6ff6ff10b9f4994a0f948a987a66cc82f98451e1).

References

Book
1. Abeles M
(1991) Corticonics
Cambridge University Press.
https://doi.org/10.1017/CBO9780511574566
- Google Scholar
(2013) Analysis of the stabilized supralinear network
Neural Computation 25:1994–2037.
https://doi.org/10.1162/NECO_a_00472
- PubMed
- Google Scholar
1. Amit DJ
2. Brunel N
(1997) Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex
Cerebral Cortex 7:237–252.
https://doi.org/10.1093/cercor/7.3.237
- PubMed
- Google Scholar
(2020) Nonlinear stimulus representations in neural circuits with approximate excitatory-inhibitory balance
PLOS Computational Biology 16:1008192.
https://doi.org/10.1371/journal.pcbi.1008192
- PubMed
- Google Scholar
1. Barkai E
2. Hasselmo ME
(1994) Modulation of the input/output function of rat piriform cortex pyramidal cells
Journal of Neurophysiology 72:644–658.
https://doi.org/10.1152/jn.1994.72.2.644
- PubMed
- Google Scholar
1. Benda J
2. Herz AVM
(2003) A universal model for spike-frequency adaptation
Neural Computation 15:2523–2564.
https://doi.org/10.1162/089976603322385063
- PubMed
- Google Scholar
(2003) Dopamine gates LTP induction in lateral amygdala by suppressing feedforward inhibition
Nature Neuroscience 6:587–592.
https://doi.org/10.1038/nn1058
- PubMed
- Google Scholar
1. Bolding KA
2. Franks KM
(2018) Recurrent cortical circuits implement concentration-invariant odor coding
Science 361:eaat6904.
https://doi.org/10.1126/science.aat6904
- PubMed
- Google Scholar
1. Bolding KA
2. Nagappan S
3. Han BX
4. Wang F
5. Franks KM
(2020) Recurrent circuitry is required to stabilize piriform cortex odor representations across brain states
eLife 9:e53125.
https://doi.org/10.7554/eLife.53125
- PubMed
- Google Scholar
1. Bondanelli G
2. Ostojic S
(2020) Coding with transient trajectories in recurrent neural networks
PLOS Computational Biology 16:e1007655.
https://doi.org/10.1371/journal.pcbi.1007655
- PubMed
- Google Scholar
1. Brette R
2. Gerstner W
(2005) Adaptive exponential integrate-and-fire model as an effective description of neuronal activity
Journal of Neurophysiology 94:3637–3642.
https://doi.org/10.1152/jn.00686.2005
- PubMed
- Google Scholar
1. Carrillo-Reid L
2. Yang W
3. Bando Y
4. Peterka DS
5. Yuste R
(2016) Imprinting and recalling cortical ensembles
Science 353:691–694.
https://doi.org/10.1126/science.aaf7560
- PubMed
- Google Scholar
1. Cayco-Gajic NA
2. Silver RA
(2019) Re-evaluating Circuit Mechanisms Underlying Pattern Separation
Neuron 101:584–602.
https://doi.org/10.1016/j.neuron.2019.01.044
- PubMed
- Google Scholar
Preprint
(2021) Regimes and Mechanisms of Transient Amplification in Abstract and Biological Networks
bioRxiv.
https://doi.org/10.1101/2021.04.01.437964
- Google Scholar
(2012) Neural population dynamics during reaching
Nature 487:51–56.
https://doi.org/10.1038/nature11129
- PubMed
- Google Scholar
(2000) Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model
Cerebral Cortex 10:910–923.
https://doi.org/10.1093/cercor/10.9.910
- PubMed
- Google Scholar
1. Cossell L
2. Iacaruso MF
3. Muir DR
4. Houlton R
5. Sader EN
6. Ko H
7. Hofer SB
8. Mrsic-Flogel TD
(2015) Functional organization of excitatory synaptic strength in primary visual cortex
Nature 518:399–403.
https://doi.org/10.1038/nature14182
- PubMed
- Google Scholar
(2007) Synaptic basis for intense thalamocortical activation of feedforward inhibitory cells in neocortex
Nature Neuroscience 10:462–468.
https://doi.org/10.1038/nn1861
- PubMed
- Google Scholar
1. Denève S
2. Machens CK
(2016) Efficient codes and balanced networks
Nature Neuroscience 19:375–382.
https://doi.org/10.1038/nn.4243
- PubMed
- Google Scholar
(2003) Binary spiking in auditory cortex
The Journal of Neuroscience 23:7940–7949.
https://doi.org/10.1523/JNEUROSCI.23-21-07940.2003
- PubMed
- Google Scholar
(2020) Cortical-like dynamics in recurrent circuits optimized for sampling-based probabilistic inference
Nature Neuroscience 23:1138–1149.
https://doi.org/10.1038/s41593-020-0671-1
- PubMed
- Google Scholar
(2021) An emergent neural coactivity code for dynamic memory
Nature Neuroscience 24:694–704.
https://doi.org/10.1038/s41593-021-00820-w
- PubMed
- Google Scholar
(2001) Dynamic predictions: oscillations and synchrony in top-down processing
Nature Reviews Neuroscience 2:704–716.
https://doi.org/10.1038/35094565
- PubMed
- Google Scholar
(2001) Efficiency and ambiguity in an adaptive neural code
Nature 412:787–792.
https://doi.org/10.1038/35090500
- PubMed
- Google Scholar
(2011) Recurrent circuitry dynamically shapes the activation of piriform cortex
Neuron 72:49–56.
https://doi.org/10.1016/j.neuron.2011.08.020
- PubMed
- Google Scholar
(2001) Categorical representation of visual stimuli in the primate prefrontal cortex
Science 291:312–316.
https://doi.org/10.1126/science.291.5502.312
- PubMed
- Google Scholar
(2007) A synaptic memory trace for cortical receptive field plasticity
Nature 450:425–429.
https://doi.org/10.1038/nature06289
- PubMed
- Google Scholar
(1989) Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex
Journal of Neurophysiology 61:331–349.
https://doi.org/10.1152/jn.1989.61.2.331
- PubMed
- Google Scholar
(2017) Short-term depression and transient memory in sensory cortex
Journal of Computational Neuroscience 43:273–294.
https://doi.org/10.1007/s10827-017-0662-8
- PubMed
- Google Scholar
(2020) Characteristics of sequential activity in networks with temporally asymmetric Hebbian learning
PNAS 117:29948–29958.
https://doi.org/10.1073/pnas.1918674117
- PubMed
- Google Scholar
(2018) Specific synaptic input strengths determine the computational properties of excitation-inhibition integration in a sound localization circuit
The Journal of Physiology 596:4945–4967.
https://doi.org/10.1113/JP276012
- PubMed
- Google Scholar
1. Goldman MS
(2009) Memory without feedback in a neural network
Neuron 61:621–634.
https://doi.org/10.1016/j.neuron.2008.12.012
- PubMed
- Google Scholar
(2003) Organization of cell assemblies in the hippocampus
Nature 424:552–556.
https://doi.org/10.1038/nature01834
- PubMed
- Google Scholar
1. Harris KD
(2005) Neural signatures of cell assembly organization
Nature Reviews Neuroscience 6:399–407.
https://doi.org/10.1038/nrn1669
- PubMed
- Google Scholar
Book
1. Hebb DO
(1949)
The Organization of Behavior: A Neuropsychological Theory

Wiley.
- Google Scholar
(2012) Non-normal amplification in random balanced neuronal networks
Physical Review E 86:1–12.
https://doi.org/10.1103/PhysRevE.86.011909
- PubMed
- Google Scholar
(2014) Optimal control of transient dynamics in balanced networks supports generation of complex movements
Neuron 82:1394–1406.
https://doi.org/10.1016/j.neuron.2014.04.045
- PubMed
- Google Scholar
(2017) Inhibitory Plasticity: Balance, Control, and Codependence
Annual Review of Neuroscience 40:557–579.
https://doi.org/10.1146/annurev-neuro-072116-031005
- PubMed
- Google Scholar
(2018) The Dynamical regime of sensory cortex: stable dynamics around a single stimulus-tuned attractor account for patterns of noise variability
Neuron 98:846–860.
https://doi.org/10.1016/j.neuron.2018.04.017
- PubMed
- Google Scholar
1. Hopfield JJ
(1982) Neural networks and physical systems with emergent collective computational abilities
PNAS 79:2554–2558.
https://doi.org/10.1073/pnas.79.8.2554
- PubMed
- Google Scholar
Book
1. Horn RA
2. Johnson CR
(1985) Matrix Analysis
Cambridge University Press.
https://doi.org/10.1017/CBO9780511810817
- Google Scholar
Book
1. Irving RS
(2004) Integers, Polynomials, and Rings: A Course in Algebra
Springer.
https://doi.org/10.1007/b97633
- Google Scholar
1. Ji XY
2. Zingg B
3. Mesik L
4. Xiao Z
5. Zhang LI
6. Tao HW
(2016) Thalamocortical Innervation Pattern in Mouse Auditory and Visual Cortex: Laminar and Cell-Type Specificity
Cerebral Cortex 26:2612–2625.
https://doi.org/10.1093/cercor/bhv099
- PubMed
- Google Scholar
1. Josselyn SA
2. Tonegawa S
(2020) Memory engrams: Recalling the past and imagining the future
Science 367:eaaw4325.
https://doi.org/10.1126/science.aaw4325
- PubMed
- Google Scholar
(2011) Functional specificity of local synaptic connections in neocortical networks
Nature 473:87–91.
https://doi.org/10.1038/nature09880
- PubMed
- Google Scholar
1. Kraynyukova N
2. Tchumatchenko T
(2018) Stabilized supralinear network can give rise to bistable, oscillatory, and persistent activity
PNAS 115:3464–3469.
https://doi.org/10.1073/pnas.1700080115
- PubMed
- Google Scholar
1. Large AM
2. Vogler NW
3. Mielo S
4. Oswald AMM
(2016) Balanced feedforward inhibition and dominant recurrent inhibition in olfactory cortex
PNAS 113:2276–2281.
https://doi.org/10.1073/pnas.1519295113
- PubMed
- Google Scholar
(2007) Dynamical synapses causing self-organized criticality in neural networks
Nature Physics 3:857–860.
https://doi.org/10.1038/nphys758
- Google Scholar
1. Litwin-Kumar A
2. Doiron B
(2012) Slow dynamics and high variability in balanced cortical networks with clustered connections
Nature Neuroscience 15:1498–1505.
https://doi.org/10.1038/nn.3220
- PubMed
- Google Scholar
1. Loebel A
2. Tsodyks M
(2002) Computation by ensemble synchronization in recurrent networks with synaptic depression
Journal of Computational Neuroscience 13:111–124.
https://doi.org/10.1023/a:1020110223441
- PubMed
- Google Scholar
(2007) Processing of sounds by population spikes in a model of primary auditory cortex
Frontiers in Neuroscience 1:197–209.
https://doi.org/10.3389/neuro.01.1.1.015.2007
- PubMed
- Google Scholar
(1998) Differential signaling via the same axon of neocortical pyramidal neurons
PNAS 95:5323–5328.
https://doi.org/10.1073/pnas.95.9.5323
- PubMed
- Google Scholar
1. Marshel JH
2. Kim YS
3. Machado TA
4. Quirin S
5. Benson B
6. Kadmon J
7. Raja C
8. Chibukhchyan A
9. Ramakrishnan C
10. Inoue M
11. Shane JC
12. McKnight DJ
13. Yoshizawa S
14. Kato HE
15. Ganguli S
16. Deisseroth K
(2019) Cortical layer-specific critical dynamics triggering perception
Science 365:eaaw5202.
https://doi.org/10.1126/science.aaw5202
- PubMed
- Google Scholar
1. Mazor O
2. Laurent G
(2005) Transient dynamics versus fixed points in odor representations by locust antennal lobe projection neurons
Neuron 48:661–673.
https://doi.org/10.1016/j.neuron.2005.09.032
- PubMed
- Google Scholar
(2019) Expectation-induced modulation of metastable activity underlies faster coding of sensory stimuli
Nature Neuroscience 22:787–796.
https://doi.org/10.1038/s41593-019-0364-9
- PubMed
- Google Scholar
Preprint
1. Miller KD
2. Palmigiano A
(2020) Generalized Paradoxical Effects in Excitatory/Inhibitory Networks
bioRxiv.
https://doi.org/10.1101/2020.10.13.336727
- Google Scholar
(2018) Sensory experience inversely regulates feedforward and feedback excitation-inhibition ratio in rodent visual cortex
eLife 7:e38846.
https://doi.org/10.7554/eLife.38846
- PubMed
- Google Scholar
(2008) Synaptic theory of working memory
Science 319:1543–1546.
https://doi.org/10.1126/science.1150769
- PubMed
- Google Scholar
(2012) Bistability and spatiotemporal irregularity in neuronal networks with nonlinear synaptic transmission
Physical Review Letters 108:158101.
https://doi.org/10.1103/PhysRevLett.108.158101
- PubMed
- Google Scholar
1. Murphy BK
2. Miller KD
(2009) Balanced amplification: a new mechanism of selective amplification of neural activity patterns
Neuron 61:635–648.
https://doi.org/10.1016/j.neuron.2009.02.005
- PubMed
- Google Scholar
1. Niessing J
2. Friedrich RW
(2010) Olfactory pattern classification by discrete neuronal network states
Nature 465:47–52.
https://doi.org/10.1038/nature08961
- PubMed
- Google Scholar
1. Okun M
2. Lampl I
(2008) Instantaneous correlation of excitation and inhibition during ongoing and sensory-evoked activities
Nature Neuroscience 11:535–537.
https://doi.org/10.1038/nn.2105
- PubMed
- Google Scholar
1. Ozeki H
2. Finn IM
3. Schaffer ES
4. Miller KD
5. Ferster D
(2009) Inhibitory stabilization of the cortical network underlies visual surround suppression
Neuron 62:578–592.
https://doi.org/10.1016/j.neuron.2009.03.028
- PubMed
- Google Scholar
1. Pala A
2. Petersen CCH
(2015) In vivo measurement of cell-type-specific synaptic connectivity and synaptic transmission in layer 2/3 mouse barrel cortex
Neuron 85:68–75.
https://doi.org/10.1016/j.neuron.2014.11.025
- PubMed
- Google Scholar
(2020) Recurrent interactions in local cortical circuits
Nature 579:256–259.
https://doi.org/10.1038/s41586-020-2062-x
- PubMed
- Google Scholar
1. Plenz D
2. Thiagarajan TC
(2007) The organizing principles of neuronal avalanches: cell assemblies in the cortex?
Trends in Neurosciences 30:101–110.
https://doi.org/10.1016/j.tins.2007.01.005
- PubMed
- Google Scholar
(2013) Stimulus-dependent variability and noise correlations in cortical MT neurons
PNAS 110:13162–13167.
https://doi.org/10.1073/pnas.1300098110
- PubMed
- Google Scholar
1. Pozzorini C
2. Naud R
3. Mensi S
4. Gerstner W
(2013) Temporal whitening by power-law adaptation in neocortical neurons
Nature Neuroscience 16:942–948.
https://doi.org/10.1038/nn.3431
- PubMed
- Google Scholar
(2004) The contribution of spike threshold to the dichotomy of cortical simple and complex cells
Nature Neuroscience 7:1113–1122.
https://doi.org/10.1038/nn1310
- PubMed
- Google Scholar
1. Romo R
2. Brody CD
3. Hernández A
4. Lemus L
(1999) Neuronal correlates of parametric working memory in the prefrontal cortex
Nature 399:470–473.
https://doi.org/10.1038/20939
- PubMed
- Google Scholar
1. Rost T
2. Deger M
3. Nawrot MP
(2018) Winnerless competition in clustered balanced networks: inhibitory assemblies do the trick
Biological Cybernetics 112:81–98.
https://doi.org/10.1007/s00422-017-0737-7
- PubMed
- Google Scholar
(2015) The stabilized supralinear network: a unifying circuit motif underlying multi-input integration in sensory cortex
Neuron 85:402–417.
https://doi.org/10.1016/j.neuron.2014.12.026
- PubMed
- Google Scholar
(2017) Balanced excitation and inhibition are required for high-capacity, noise-robust neuronal selectivity
PNAS 114:E9366–E9375.
https://doi.org/10.1073/pnas.1705841114
- PubMed
- Google Scholar
1. Rupprecht P
2. Friedrich RW
(2018) Precise Synaptic Balance in the Zebrafish Homolog of Olfactory Cortex
Neuron 100:669–683.
https://doi.org/10.1016/j.neuron.2018.09.013
- PubMed
- Google Scholar
1. Sadeh S
2. Clopath C
(2021) Inhibitory stabilization and cortical computation
Nature Reviews Neuroscience 22:21–37.
https://doi.org/10.1038/s41583-020-00390-z
- PubMed
- Google Scholar
1. Sanzeni A
2. Akitake B
3. Goldbach HC
4. Leedy CE
5. Brunel N
6. Histed MH
(2020) Inhibition stabilization is a widespread property of cortical networks
eLife 9:e54875.
https://doi.org/10.7554/eLife.54875
- PubMed
- Google Scholar
(2021) The generation of cortical novelty responses through inhibitory plasticity
eLife 10:e65309.
https://doi.org/10.7554/eLife.65309
- PubMed
- Google Scholar
1. Shew WL
2. Clawson WP
3. Pobst J
4. Karimipanah Y
5. Wright NC
6. Wessel R
(2015) Adaptation to sensory input tunes visual cortex to criticality
Nature Physics 11:659–663.
https://doi.org/10.1038/nphys3370
- Google Scholar
(2018) A transformation from temporal to ensemble coding in a model of piriform cortex
eLife 7:e34831.
https://doi.org/10.7554/eLife.34831
- PubMed
- Google Scholar
(1997) Impaired odour discrimination on desynchronization of odour-encoding neural assemblies
Nature 390:70–74.
https://doi.org/10.1038/36335
- PubMed
- Google Scholar
1. Thorpe S
2. Fize D
3. Marlot C
(1996) Speed of processing in the human visual system
Nature 381:520–522.
https://doi.org/10.1038/381520a0
- PubMed
- Google Scholar
1. Tsodyks MV
2. Markram H
(1997) The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability
PNAS 94:719–723.
https://doi.org/10.1073/pnas.94.2.719
- PubMed
- Google Scholar
(1997) Paradoxical effects of external modulation of inhibitory interneurons
The Journal of Neuroscience 17:4382–4388.
https://doi.org/10.1523/JNEUROSCI.17-11-04382.1997
- PubMed
- Google Scholar
1. van Vreeswijk C
2. Sompolinsky H
(1996) Chaos in neuronal networks with balanced excitatory and inhibitory activity
Science 274:1724–1726.
https://doi.org/10.1126/science.274.5293.1724
- PubMed
- Google Scholar
1. van Vreeswijk C
2. Hansel D
(2001) Patterns of synchrony in neural networks with spike adaptation
Neural Computation 13:959–992.
https://doi.org/10.1162/08997660151134280
- PubMed
- Google Scholar
1. Varela JA
2. Sen K
3. Gibson J
4. Fost J
5. Abbott LF
6. Nelson SB
(1997) A quantitative description of short-term plasticity at excitatory synapses in layer 2/3 of rat primary visual cortex
The Journal of Neuroscience 17:7926–7940.
https://doi.org/10.1523/JNEUROSCI.17-20-07926.1997
- PubMed
- Google Scholar
1. Vinje WE
2. Gallant JL
(2000) Sparse coding and decorrelation in primary visual cortex during natural vision
Science 287:1273–1276.
https://doi.org/10.1126/science.287.5456.1273
- PubMed
- Google Scholar
(2011) Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks
Science 334:1569–1573.
https://doi.org/10.1126/science.1211095
- PubMed
- Google Scholar
1. Wehr M
2. Zador AM
(2003) Balanced inhibition underlies tuning and sharpens spike timing in auditory cortex
Nature 426:442–446.
https://doi.org/10.1038/nature02116
- PubMed
- Google Scholar
1. Wong KF
2. Wang XJ
(2006) A recurrent network mechanism of time integration in perceptual decisions
The Journal of Neuroscience 26:1314–1328.
https://doi.org/10.1523/JNEUROSCI.3733-05.2006
- PubMed
- Google Scholar
(2020) Homeostatic mechanisms regulate distinct aspects of cortical circuit dynamics
PNAS 117:24514–24525.
https://doi.org/10.1073/pnas.1918368117
- PubMed
- Google Scholar
Software
1. Wu YK
(2021) Nonlinear Transient Amplification, version swh:1:rev:6ff6ff10b9f4994a0f948a987a66cc82f98451e1
Software Heritage.

https://archive.softwareheritage.org/swh:1:dir:bcdf9ea652853e2f06b154735e693fa154943072;origin=https://github.com/fmi-basel/gzenke-nonlinear-transient-amplification;visit=swh:1:snp:6b78c97e1671014d095bb0db7eaa97aaebe0086b;anchor=swh:1:rev:6ff6ff10b9f4994a0f948a987a66cc82f98451e1
1. Yakovlev V
2. Fusi S
3. Berman E
4. Zohary E
(1998) Inter-trial neuronal activity in inferior temporal cortex: a putative vehicle to generate long-term visual associations
Nature Neuroscience 1:310–317.
https://doi.org/10.1038/1131
- PubMed
- Google Scholar
(2015) Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks
Nature Communications 6:1–13.
https://doi.org/10.1038/ncomms7922
- PubMed
- Google Scholar
Preprint
(2018) Functional selectivity and specific connectivity of inhibitory neurons in primary visual cortex
bioRxiv.
https://doi.org/10.1101/294835
- Google Scholar
1. Zucker RS
2. Regehr WG
(2002) Short-term synaptic plasticity
Annual Review of Physiology 64:355–405.
https://doi.org/10.1146/annurev.physiol.64.092501.114547
- PubMed
- Google Scholar

Decision letter

Timothy O'Leary

Reviewing Editor; University of Cambridge, United Kingdom
Ronald L Calabrese

Senior Editor; Emory University, United States

Our editorial process produces two outputs: (i) public reviews designed to be posted alongside the preprint for the benefit of readers; (ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.

Decision letter after peer review:

Thank you for submitting your article "Nonlinear transient amplification in recurrent neural networks with short-term plasticity" for consideration by eLife. Your article has been reviewed by 2 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Ronald Calabrese as the Senior Editor. The reviewers have opted to remain anonymous.

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

1) The findings in this work need to be better contextualised with existing literature. Some of the claims and results are presaged in other work and the reviewers agreed that the manuscript would benefit from better acknowledgement and discussion of prior work – see reviewer comments for details.

2) There are concerns that the behaviour of the model violates experimental observations (Reviewer 1, comment 1). The authors should address this concern, with additional results or sound arguments that reconcile these observations with the model.

3) Greater depth in the intuition and analysis of the stabilisation mechanism in this network is needed to justify all of the conclusions (Reviewer 1, comments 3-4; Reviewer 2, comments 4-6). This will also elevate the contribution of the manuscript and allow new findings to be better appreciated in the context of previous work.

4) The reviewers provide detailed comments below for improving the clarity and depth of the presentation. The authors should consider and respond to these suggestions.

Reviewer #1 (Recommendations for the authors):

– Line 51. Results from [Bolding and Franks, Science 2018] have been explained in [Stern et al., eLife 2018], without the need of adaptation. This work should be cited here, as it provides an explanation of transient response that is alternative to the mechanism proposed by the authors.

– Line 55. Isn't coupling strength, rather than inhibition, the key factor that linearizes network responses (e.g. see [van Vreeswijk and Sompolinsky, Neural computation 1998]).

– Above line 101. I do not understand why strong E-E connectivity is consistent only with detJ<0. Strong E-E connectivity typically also requires strong E-I and I-E connectivity [Tsodyks et al., J neuroscience 1997]. The assumption of detJ<0 seems important for the behavior of the characteristic function F(z). How are the results of the paper modified in the case in which det J>0?

– Line 124-126. I am confused about the conclusion of this section. Results discussed in Pages 5 and 6 are relative to the stability of the network as a function of input strength; why is this relevant for the nonlinear amplification of inputs? Nonlinear amplification of inputs is a general property of the SSN, which emerges also for parameters that lead to stable dynamics [Ahmadian et al., Neural computation 2013]. The authors should also explain why they select parameters that lead to unstable responses for large inputs. On a related note, the fact that the supralinear input-output function leads to activity dependent coupling and, for certain parameters, to unstable dynamics has been shown in [Ahmadian et al., Neural computation 2013]; the authors should acknowledge this fact.

– Line 188. There are many more papers related to ISN in cortex to be cited; see [Sadeh and Clopath, Nat Rev Neurosci 2021] for a summary.

– Line 273-274. It would be useful to include citations of experiments supporting this claim.

– Line 407-409. What is the relation of this work with network criticality?

– The stability of a system with excitatory-inhibitory neurons and STD depends on the eigenvalues of a three dimensional matrix, why does the condition described after Equation (32) do not include the component given by the dynamics of x?

Reviewer #2 (Recommendations for the authors):

First a note about the model of Loebel et al.,: note that while single neurons in their models have a ReLU nonlinearity, they receive different external baseline inputs, and therefore the population has an effective input-output nonlinearity that is convex and supralinear as in the current manuscript.

Specific/detailed comments:

1 – In the abstract, intro and discussion it is claimed that nonnormal connectivity is in clash with symmetric connectivity the E-subnetwork (and thus with the Hebbian learning/doctrine). But non-normal transient amplification happens in E-I networks in which the E-E part of connectivity matrix is indeed symmetric, while the full matrix is necessarily asymmetric and nonnormal (due to Dale's law). So the symmetry of the E-E connectivity (the part that is affected by Hebbian plasticity in most models) is in fact consistent with non-normal TA.

2 – Similarly in lines 56-65 of Intro (and 438-454 of discussion) it is claimed that balanced amplification (due to feedback inhibition and recurrent excitation) cannot generate strong TA, because "several ensembles need to be chained together into a hidden feedforward structure" which the authors say is "at odds with the often observed symmetric excitatory connectivity". But again symmetric E-E connectivity is consistent with nonnormality of the full (E-I) connectivity matrix, and moreover the chains are hidden between patterns, and not anatomical between neuronal ensembles. Also long hidden chains are not necessary, and when recurrent E and I connections are strong, short chains (e.g. in the SSN model) can also yield strong enough TA to be consistent with those observed in sensory cortex – though probably not as strong as in NTA. This parts should thus be more accurately written and if the claim is maintained a more thorough comparison (e.g. in a supplementary figure) with a recurrent network with neural nonlinearity (e.g. the supralinear powerlaw as in the current model) but without STP (but possibly with different connectivity so that the instability in Figure 1 does not happen) should be provided.

3 – Given that property C is a main selling point, they should characterize it better. The only result plot about this is currently Figure 1F, in which activity actually diverges post-stimulus. The nonlinear threshold (for stimulus strength) should be quantified and shown in some plot (e.g. to be added to Figure 2) for the case in which STP re-stabilizes activity. In particular the dependence of NTA (e.g. as measured by the ratio of transient peak to steady-state response) on parameters such as stimulus strength (c.f. Figure 4B of Loebel and Tsodyks 2002) or J_EE, can be characterised/plotted.

4 – The effect of co-tuning of E-I (feature G) is characterised in abstract and the corresponding sub-section (its title on p. 11 and Figure 3 title) as broadening the parameter regime of NTA. But Figure 3 in fact shows that NTA happens even in the bistable case and with global inhibition. So a more accurate characterization his that E/I co-tuning broadens the parameter regime in which activity is unistable and non-persistent (which is suitable for sensory processing), without adversely affecting NTA. Also given that NTA is weaker -though still strong- with co-tuned inhibition, perhaps the strength of NTA can be quantatively compared between the two cases of co-tuned vs global inhibition.

5 – In the parts where intuitive explanation of re-stabilization (or lack thereof) due to SFA or STP are given on pp. 7-8, I would first of all provide Equations. 20-26 right there, or at least refer to them. More importantly when referring to F(z) they should mention that they are employing a fast-slow approximation using the separation of timescales between fast neural activity and adaptation variables (a in SFA, x in STD, etc) are therefore fixing the slow variables to their value at the previous/baseline fixed point. At least I believe this is what they are doing and showing in Figure 2D, but again I'm suggesting they should write this more clearly. In particular, it seems to me that while Equation 30 (with x assumed to be fixed at its value at the preceding fixed point) is consistent with the above interpretation (which I believe is the right thing to do), Equation 28 is not, as it is obtained by setting a equal to its value at its fixed point corresponding to the dynamical value of r_E, and this is the right thing to do if "a" was a fast variable, not a slow one (as it presumably is). So, I believe, the correct F(z) for the SFA model should be rederived by assuming treating "a" in equation 20 as a constant independent of r_E (rather solving the steady state from 21 and plugging it in Equation 20).

6 – Switching to ISN with stimulus from a non-ISN baseline is presumably parameter dependent, and this dependence should be characterized better (at least give examples of networks in which this is not the case, in a supplementary figure, and correspondingly weaken the absolute statement in lines 192-194 so that it says this transition "can" happen, which would avoid implying that this is necessarily the case, assuming it's not). Also in the ISN index (Equation 13) shouldn't J_EE be multiplied by x (depression variable) at the fixed point?

7 – I found the part about pattern separation (p. 14) not clearly written and motivated. In fact I think the term "pattern separation" should be replaced with "stimulus selectivity". Pattern separation is the appropriate term when a networks output patterns are more separable than the patterns at the input-level. However, in the current case in which the external input is given only to E1 (only to one of the two E populations), the input is fully separable, so the network is potentially only capable of reducing separation. At any rate the pattern separation would be meaningful if they had compared separation at input-level (distance from decision boundary in gE1 – gE2 plane) with separation at output-level (distance from decision boundary in rE1 – rE2 plane). I think what is currently shown and charcterized is stimulus selectivity and the result is that transient onset responses are more selective than fixed point responses. So this section (and corresponding parts in intro/discussion) should be rewritten to convey this.

7a – On the other hand, this finding (stronger selectivity at onset, vs steady-state or sustained response) is presented as a desired outcome. This may be so, but it seems inconsistent with empirical findings on orientation selectivity in (mouse) visual cortex in which orientation selectivity is much stronger for sustained responses compared to onset transients. This should be mentioned (e.g. in lines 290, 326, and 418-420).

7b – The distance to decision boundary used to characterize "pattern separation" is not clearly described in lines 280-282. The description in the Methods section was also somewhat ambiguous, but giving a formula there would solve this issue. What I understood is that the measure is basically rE1 – rE2 where rE1 and rE2 are population mean activities of the E neurons in ensembles 1 and 2. If so, why not use the normalized measure (rE1 – rE2)/(rE1 + rE2)?

8 – The property B is depicted in various places (e.g. line 410) by saying NTA requires or relies on symmetric E-E connections. While NTA clearly relies on the destabilizing effect of recurrent EE connections, it is not clear that the symmetry (at least exact symmetry) of these connections is necessary. Indeed their LIF spiking net has random connectivity, so the EE part of connectivity at the level of single neurons in that network is actually not symmetric. These sentences could thus be more accurately written to clarify this point.

9 – In the current study, external stimulus was only given to excitatory neurons, with authors saying the study of feedforward input to inhibition is beyond the scope of this work, yet they also say (lines 429-432) that "we are confident that our main findings remain unaffected in the presence of substantial feedforward inhibition", and cite supplementary figure 1, which however was for the case without STP. I think it would be easy to provide a supplementary figure characterizing NTA in a network with STP in which gI is nonzero for stimulus (e.g. gI = gE).

https://doi.org/10.7554/eLife.71263.sa1

Author response

Essential revisions:

1) The findings in this work need to be better contextualised with existing literature. Some of the claims and results are presaged in other work and the reviewers agreed that the manuscript would benefit from better acknowledgement and discussion of prior work – see reviewer comments for details.

Thanks for the constructive feedback. The revised version now positions itself more clearly relative to the existing literature. Specifically, we highlight the close relation to (Loebel and Tsodyks, 2002; Loebel et al., 2007) and how we extend this work (see our reply to Point P 2.1). Moreover, we formulated the manuscript such that NTA is a possible alternative mechanism to balanced amplification (Murphy and Miller, 2009) and that the two are not mutually exclusive in biological networks (see our reply to Points P 1.3 and P 2.1). Finally, we elucidate the relation to the spiking network models without STP studied by Stern et al., (2018) (see our reply to Point P 1.6).

2) There are concerns that the behaviour of the model violates experimental observations (Reviewer 1, comment 1). The authors should address this concern, with additional results or sound arguments that reconcile these observations with the model.

We were very grateful for these comments because they prompted us to perform additional analyses that led to new insights. Importantly, we were able to show that NTA can also occur in networks that are an ISN already at baseline. This effect depends on the network parameters. These additional results reconcile the model with recent experimental findings suggesting that the visual cortex is an ISN at baseline (Sanzeni et al., 2020). However, because the effect is parameter dependent, it leaves the possibility open that other sensory areas may be a non-ISN at baseline, e.g., olfactory areas (see our reply to Point P 1.1). Moreover, we also show that the firing rates of unstimulated ensemble neurons are not necessarily suppressed when stimulating the rest of the ensemble which renders our model compatible with observations on recent pattern completion experiments (Carrillo-Reid et al., 2016; Marshel et al., 2019). Again, this effect is parameter-dependent and our analysis suggests that it also changes with the stimulus intensity, which also makes a prediction that could be tested experimentally (see our reply to Point P 1.2).

3) Greater depth in the intuition and analysis of the stabilisation mechanism in this network is needed to justify all of the conclusions (Reviewer 1, comments 3-4; Reviewer 2, comments 4-6). This will also elevate the contribution of the manuscript and allow new findings to be better appreciated in the context of previous work.

Again thanks to this feedback, we deepened our understanding of the mechanisms through additional analyses to further justify our conclusions. This work led to the inclusion of multiple additional figure supplements and Methods sections. We also submit an Appendix with this revised manuscript. We addressed the reviewer concerns, specifically to Points P 1.4, P 1.5, P 2.5, P 2.6, and P 2.7, below.

4) The reviewers provide detailed comments below for improving the clarity and depth of the presentation. The authors should consider and respond to these suggestions.

Many thanks for the thoughtful feedback. Please find our responses to these points below.

Reviewer #1 (Recommendations for the authors):

– Line 51. Results from [Bolding and Franks, Science 2018] have been explained in [Stern et al., eLife 2018], without the need of adaptation. This work should be cited here, as it provides an explanation of transient response that is alternative to the mechanism proposed by the authors.

Thanks for the pointer. We now added Stern et al., (2018) in the Discussion. The main difference in their work is that the networks they study have feedback inhibition that is strong enough to stabilize dynamics already in the absence of STP. We now cite it as follows:

“In addition, in spiking neural networks, strong input can induce synchronous firing at the population level which is subsequently stabilized by strong feedback inhibition without the requirement for STP mechanisms (Stern et al., 2018)”

– Line 55. Isn’t coupling strength, rather than inhibition, the key factor that linearizes network responses (e.g. see [van Vreeswijk and Sompolinsky, Neural computation 1998]).

Thanks for raising this point. It is both the negative feedback from inhibition and strong coupling strength. To clarify this point, we highlight the feedback inhibition is powerful and strong in the introduction which now reads:

– Above line 101. I do not understand why strong E-E connectivity is consistent only with detJ<0. Strong E-E connectivity typically also requires strong E-I and I-E connectivity [Tsodyks et al., J neuroscience 1997]. The assumption of detJ<0 seems important for the behavior of the characteristic function F(z). How are the results of the paper modified in the case in which det J>0?

Thanks for asking this question. We realized that we were not clear about this point in our introduction. We are, specifically, interested in networks with detJ < 0 and supralinear activation functions because they have the ability to generate run-away dynamics for large enough initial conditions (Ahmadian et al., 2013) (and in the absence of STP). This is a key ingredient of NTA and it underlies its threshold dependence and high amplification gain, but it requires an additional “reset” or re-stabilization mechanism which is implemented by STP. Previous work has largely focused on the case of detJ > 0 because this ensures network stability for any input provided that inhibition is fast enough, in particular, when it is infinitely fast (_I∕_E → 0).

To make this point clear we rewrote the corresponding Results section following Equation (3). It now reads as follows:

“We were specifically interested in networks with strong recurrent excitation that can generate positive feedback dynamics in response to external inputs g_E. Therefore, we studied networks with

det(J) = −J_EEJ_II + J_IEJ_EI < 0 . (1)

In contrast, networks in which recurrent excitation is met by strong feedback inhibition such that det(J) > 0 are unable to generate positive feedback dynamics provided that inhibition is fast enough (Ahmadian et al., 2013).”

It is worth noting, however, that positive runaway dynamics can also occur in networks with detJ > 0 if inhibition is not fast enough. This, for instance, is shown in the example below with E = 20ms and _I = 60ms, in which additional stimulation can still lead to unstable network dynamics in networks with det(J) > 0. The conclusions about the stabilization mechanisms, however, still seem to hold as we now show in a new figure supplement with simulations (Figure 2—figure supplement 11). We now clearly explain this point and refer to the new figure supplement in the Discussion section, where we write:

“NTA’s requirement to generate positive feedback dynamics through recurrent excitation, motivated our focus on networks with det(J) < 0. As demonstrated in previous work (Ahmadian et al., 2013), supralinear networks with det(J) > 0 and instantaneous inhibition (_I∕_E→ 0) are always stable for any given input, they are thus unable to generate positive feedback dynamics. In addition, networks with det(J) > 0 can exhibit a range of interesting behaviors, e.g., oscillatory dynamics and persistent activity (Kraynyukova and Tchumatchenko, 2018). It is worth noting, however, that for delayed or slow inhibition, stimulation can still lead to unstable network dynamics in networks with det(J) > 0. Nevertheless, our simulations suggest that our main conclusions about the stabilization mechanisms still hold (Figure 2—figure supplement 11).”

– Line 124-126. I am confused about the conclusion of this section. Results discussed in Pages 5 and 6 are relative to the stability of the network as a function of input strength; why is this relevant for the nonlinear amplification of inputs? Nonlinear amplification of inputs is a general property of the SSN, which emerges also for parameters that lead to stable dynamics [Ahmadian et al., Neural computation 2013]. The authors should also explain why they select parameters that lead to unstable responses for large inputs. On a related note, the fact that the supralinear input-output function leads to activity dependent coupling and, for certain parameters, to unstable dynamics has been shown in [Ahmadian et al., Neural computation 2013]; the authors should acknowledge this fact.

Our apologies, this was not explained clearly in our previous submission. We agree that some form of nonlinear amplification can also occur in SSNs, which are stabilized through inhibition by design, as also shown in (Echeveste et al., 2020). However, in the present manuscript we consider networks which can transiently destabilize due to the temporary lack of inhibition, which is essentially the definition of NTA. NTA relies on the runway dynamics for amplification, but renders them transient through the addition of STP. Essentially the same mechanism underlies neuronal action potential generation whereby depolarization of the membrane potential leads to the opening of sodium channels, which leads to further depolarization, which leads to further opening … If it wasn’t for the inactivation of sodium channels and the simultaneous opening of potassium channels which induce the reset, the spike would never “finish”. We fully agree with the reviewer, that supralinear input-output functions lead to activity-dependent coupling and associated input-dependent instabilities was shown in (Ahmadian et al., 2013). To clarify this point and to also explain why we chose parameters that lead to unstable responses of large inputs we now clearly state these points in the beginning of our Results section just below Equation (3):

“We were specifically interested in networks with strong recurrent excitation that can generate positive feedback dynamics in response to external inputs . Therefore, we studied networks with

det(J) = −J_EEJ_II + J_IEJ_EI < 0 . (2)

– Line 188. There are many more papers related to ISN in cortex to be cited; see [Sadeh and Clopath, Nat Rev Neurosci 2021] for a summary.

Thank you for this comment. We have now included additional references in the sentence in line 193-195.

“Recent studies suggest that cortical networks operate as inhibition-stabilized networks (ISNs) (Sanzeni et al., 2020; Sadeh and Clopath, 2021), in which the excitatory network is unstable in the absence of feedback inhibition (Tsodyks et al., 1997; Ozeki et al., 2009).”

– Line 273-274. It would be useful to include citations of experiments supporting this claim.

Thank you for this comment. We have now added the corresponding references supporting the claim in line 310-312.

“Neural circuits are capable of generating stereotypical activity patterns in response to partial cues and forming distinct representations in response to different stimuli (Carrillo-Reid et al., 2016; Marshel et al., 2019; Bolding et al., 2020; Vinje and Gallant, 2000; Cayco-Gajic and Silver, 2019).”

– Line 407-409. What is the relation of this work with network criticality?

That’s an interesting question. There are some commonalities, but also some differences. In network criticality, and specifically in self-organized criticality the network is at a critical point at which a phase transition occurs, which leads the system displays hallmarks of spatial and temporal scale-invariance, e.g., as stimulus responses on a diversity of time-scales and power-law neuronal avalanche distributions. To that end, the network model is typically kept at the edge of chaos and driven by external noise. In contrast, in this article we studied networks which are for most part kept away from any critical point, i.e., from the transition between stable and transiently unstable dynamics (unless when the stimulus is turned on, but rather away). It is, however, conceivable that for suitable input statistics the boundary between the two becomes blurred. To elucidate this point, we added three sentences in the Discussion section which read as follows:

“NTA shares some properties with the notion of network criticality in the brain, like synchronous activation of cell ensembles (Plenz and Thiagarajan, 2007) and STP which can tune networks to a critical state (Levina et al., 2007). However, in contrast to most models of criticality, in NTA an ensemble briefly transitions to supercritical dynamics in a controlled, stimulus-dependent manner rather than spontaneously. Yet, how the two paradigms are connected at a more fundamental level, is an intriguing question left for future work.”

– The stability of a system with excitatory-inhibitory neurons and STD depends on the eigenvalues of a three dimensional matrix, why does the condition described after Equation (32) do not include the component given by the dynamics of x?

We thank the reviewer for this comment. The relevant equations now include. Specifically, we have redone substantial parts of our analysis in response to Point P 1.5. In particular, we now added a detailed analysis for the 3D system which incorporates the components corresponding to the dynamics of the STP variables. To that end, we added several new method sections in which we performed linear stability analysis on the full 3D systems. The derivations are laid out in the following Methods and Appendix sections:

– Stability conditions in networks with SFA (Methods)

– Stability conditions in networks with E-to-E STD (Methods)

– Stability conditions in networks with E-to-I STF (Appendix 1)

Reviewer #2 (Recommendations for the authors):

First a note about the model of Loebel et al.,: note that while single neurons in their models have a ReLU nonlinearity, they receive different external baseline inputs, and therefore the population has an effective input-output nonlinearity that is convex and supralinear as in the current manuscript.

Specific/detailed comments:

1 – In the abstract, intro and discussion it is claimed that nonnormal connectivity is in clash with symmetric connectivity the E-subnetwork (and thus with the Hebbian learning/doctrine). But non-normal transient amplification happens in E-I networks in which the E-E part of connectivity matrix is indeed symmetric, while the full matrix is necessarily asymmetric and nonnormal (due to Dale's law). So the symmetry of the E-E connectivity (the part that is affected by Hebbian plasticity in most models) is in fact consistent with non-normal TA.

We fully agree with the reviewer that in both cases, NTA and balanced amplification, the connectivity matrices are non-normal and that in our original submission this was not described accurately. To clarify this point, we extensively revised the manuscript in several places and, specifically, in the passages where we refer to balanced amplification (see also our reply to Point P 1.3).

First, we revised the Abstract and removed the sentence claiming a “clash” between nonnormal connectivity and symmetric connectivity in the E-subnetwork:

“To rapidly process information, neural circuits have to amplify specific activity patterns transiently. How the brain performs this nonlinear operation remains elusive. Hebbian assemblies are one possibility whereby strong recurrent excitatory connections boost neuronal activity. However, such Hebbian amplification is often associated with dynamical slowing of network dynamics, non-transient attractor states, and pathological run-away activity. Feedback inhibition can alleviate these effects but typically linearizes responses and reduces amplification gain. Here we study NTA, a plausible alternative mechanism that reconciles strong recurrent excitation with rapid amplification while avoiding the above issues. NTA has two distinct temporal phases. Initially, positive feedback excitation selectively amplifies inputs that exceed a critical threshold. Subsequently, short-term plasticity quenches the run-away dynamics into an inhibition-stabilized network state. By characterizing NTA in supralinear network models, we establish that the resulting onset transients are stimulus selective and well-suited for speedy information processing. Further, we find that excitatory-inhibitory co-tuning widens the parameter regime in which NTA is possible in the absence of persistent activity. In summary, NTA provides a parsimonious explanation for how excitatory-inhibitory co-tuning and short-term plasticity collaborate in recurrent networks to achieve transient amplification.”

In the revised manuscript, we write in the Introduction:

“Preventing run-away excitation and multi-stable attractor dynamics in recurrent networks requires powerful and often finely tuned feedback inhibition resulting in EI balance (Amit and Brunel, 1997; Compte et al., 2000; Litwin-Kumar and Doiron, 2012; Ponce-Alvarez et al., 2013; Mazzucato et al., 2019), which tends to linearize steady-state activity (Van Vreeswijk and Sompolinsky, 1996; Baker et al., 2020). Murphy and Miller (2009) proposed balanced amplification which reconciles transient amplification with strong recurrent excitation by tightly balancing recurrent excitation with strong feedback inhibition (Goldman, 2009; Hennequin et al., 2012, 2014; Bondanelli and Ostojic, 2020; Gillett et al., 2020). Importantly, balanced amplification was formulated for linear network models of excitatory and inhibitory neurons. Due to linearity, it intrinsically lacks the ability to nonlinearly amplify stimuli which limits its capabilities for pattern completion and pattern separation. Further, how balanced amplification relates to nonlinear neuronal activation functions and nonlinear synaptic transmission as, for instance, mediated by STP (Tsodyks and Markram, 1997; Markram et al., 1998; Zucker and Regehr, 2002; Pala and Petersen, 2015), remains elusive. This begs the question of whether there are alternative nonlinear amplification mechanisms and how they relate to existing theories of recurrent neural network processing.”

Finally, in the Discussion where we compare to previous work, we write:

Several theoretical studies approached the problem of transient amplification in recurrent neural network models. Loebel and Tsodyks (2002) have described an NTA-like mechanism as a driver for powerful ensemble synchronization in rate-based networks and in spiking neural network models of auditory cortex (Loebel et al., 2007). Here we generalized this work to both E-to-E STD and E-to-I STF and provide an in-depth characterization of its amplification capabilities, pattern completion properties, and the resulting network states with regard to their inhibition-stabilization properties. Moreover, we showed that SFA cannot provide similar network stabilization and explored how EI co-tuning interacts with NTA. Finally, we contrasted NTA to alternative transient amplification mechanisms. Balanced amplification is a particularly well-studied transient amplification mechanism (Murphy and Miller, 2009; Goldman, 2009; Hennequin et al., 2014; Bondanelli and Ostojic, 2020; Gillett et al., 2020; Christodoulou et al., 2021) that relies on non-normality of the connectivity matrix to selectively and rapidly amplify stimuli. Importantly, balanced amplification occurs in networks in which strong recurrent excitation is appropriately balanced by strong recurrent inhibition. It is capable of generating rich transient activity in linear network models (Hennequin et al., 2014), and selectively amplifies specific activity patterns, but without a specific activation threshold. In addition, in spiking neural networks, strong input can induce synchronous firing at the population level which is subsequently stabilized by strong feedback inhibition without the requirement for STP mechanisms (Stern et al., 2018). These properties contrast with NTA,

which has a nonlinear activation threshold and intrinsically relies on STP to stabilize otherwise unstable run-away dynamics. Due to the switch of the network’s dynamical state, NTA’s amplification can be orders of magnitudes larger than balanced amplification (Figure 2—figure supplement 3). Interestingly, after the transient amplification phase, ensemble dynamics settle in an inhibitory-stabilized state, which renders NTA compatible with previous work on SSNs but in the presence of STP. Finally, although NTA and balanced amplification rely on different amplification mechanisms, they are not mutually exclusive and could, in principle, co-exist in biological networks.”

2 – Similarly in lines 56-65 of Intro (and 438-454 of discussion) it is claimed that balanced amplification (due to feedback inhibition and recurrent excitation) cannot generate strong TA, because "several ensembles need to be chained together into a hidden feedforward structure" which the authors say is "at odds with the often observed symmetric excitatory connectivity". But again symmetric E-E connectivity is consistent with nonnormality of the full (E-I) connectivity matrix, and moreover the chains are hidden between patterns, and not anatomical between neuronal ensembles. Also long hidden chains are not necessary, and when recurrent E and I connections are strong, short chains (e.g. in the SSN model) can also yield strong enough TA to be consistent with those observed in sensory cortex – though probably not as strong as in NTA. This parts should thus be more accurately written and if the claim is maintained a more thorough comparison (e.g. in a supplementary figure) with a recurrent network with neural nonlinearity (e.g. the supralinear powerlaw as in the current model) but without STP (but possibly with different connectivity so that the instability in Figure 1 does not happen) should be provided.

We agree with the reviewer, we now removed the misleading explanation pertaining to the “chains” in the Introduction and rewrote the corresponding section more accurately. The relevant text passages are bold faced in our reply to the previous point (Point P 2.2). Moreover, to characterize the difference in transient amplification between the different mechanisms, we added Figure 2—figure supplement 3 which shows the amplification index in linear networks and SNNs in comparison to networks exhibiting NTA. Of course, a direct comparison is difficult because the mechanisms are parameter dependent. In particular, in Figure 2—figure supplement 3 we simulated a linear network with the same connection strengths as in the NTA network, while only changing the nonlinearity. Similarly, for the SSN network, we simply removed STP and increased J_IE and J_II by the minimal amount that would ensure stability. Even though, we only show this difference numerically, we believe that the overall results are general and the same qualitative behavior will also arise in other parameter regimes.

3 – Given that property C is a main selling point, they should characterize it better. The only result plot about this is currently Figure 1F, in which activity actually diverges post-stimulus. The nonlinear threshold (for stimulus strength) should be quantified and shown in some plot (e.g. to be added to Figure 2) for the case in which STP re-stabilizes activity. In particular the dependence of NTA (e.g. as measured by the ratio of transient peak to steady-state response) on parameters such as stimulus strength (c.f. Figure 4B of Loebel and Tsodyks 2002) or J_EE, can be characterised/plotted.

This point is well taken. We performed additional simulations to characterize the nonlinear threshold behavior of NTA and its dependence on input strength_E as well as the recurrent excitatory weight J_EE. These results are now included in the new Figure 2—figure supplement 2. We now refer to these new results in the Results section where we write:

“Crucially, transient amplification in supralinear networks with STP occurs above a critical threshold (Figure 2—figure supplement 2A, B), and requires recurrent excitation J_EE to be sufficiently strong (Figure 2—figure supplement 2C, D). To quantify the amplification ability of these networks, we calculated the ratio of the evoked peak firing rate to the input strength, henceforth called the “Amplification index”.”

4 – The effect of co-tuning of E-I (feature G) is characterised in abstract and the corresponding sub-section (its title on p. 11 and Figure 3 title) as broadening the parameter regime of NTA. But Figure 3 in fact shows that NTA happens even in the bistable case and with global inhibition. So a more accurate characterization his that E/I co-tuning broadens the parameter regime in which activity is unistable and non-persistent (which is suitable for sensory processing), without adversely affecting NTA. Also given that NTA is weaker -though still strong- with co-tuned inhibition, perhaps the strength of NTA can be quantatively compared between the two cases of co-tuned vs global inhibition.

This is a good point and we thank the reviewer for the comment. To clarify this in the revised manuscript, we updated the relevant sentence in the Abstract, renamed the corresponding section to “Co-tuned inhibition broadens the parameter regime of NTA in the absence of persistent activity,” and performed additional simulations to quantify the difference of amplification ability between networks with global inhibition and networks with co-tuned inhibition. To that end, we computed the amplification index for both architectures while keeping the total connection strength between the two the same. Networks with co-tuned inhibition are constructed such that E-to-I, I-to-E, and I-to-I connection strengths within the ensemble are two times as strong as the ones between ensembles. In contrast to networks with global inhibition, these connection strengths are the same for within the ensemble and across ensembles. This shows, as the reviewer suspected, that NTA is weaker — but still strong — in networks with co-tuned inhibition.

Finally, we now conclude the corresponding section on co-tuning as follows:

“In comparison to the ensemble with global inhibition, the ensemble with co-tuned inhibition exhibits weaker – but still strong – NTA (Figure 3—figure supplement 1). Thus, co-tuned inhibition broadens the parameter regime in which NTA is possible while simultaneously avoiding persistent attractor dynamics.”

5 – In the parts where intuitive explanation of re-stabilization (or lack thereof) due to SFA or STP are given on pp. 7-8, I would first of all provide Equations. 20-26 right there, or at least refer to them. More importantly when referring to F(z) they should mention that they are employing a fast-slow approximation using the separation of timescales between fast neural activity and adaptation variables (a in SFA, x in STD, etc) are therefore fixing the slow variables to their value at the previous/baseline fixed point. At least I believe this is what they are doing and showing in Figure 2D, but again I'm suggesting they should write this more clearly. In particular, it seems to me that while Equation 30 (with x assumed to be fixed at its value at the preceding fixed point) is consistent with the above interpretation (which I believe is the right thing to do), Equation. 28 is not, as it is obtained by setting a equal to its value at its fixed point corresponding to the dynamical value of r_E, and this is the right thing to do if "a" was a fast variable, not a slow one (as it presumably is). So, I believe, the correct F(z) for the SFA model should be rederived by assuming treating "a" in equation 20 as a constant independent of r_E (rather solving the steady state from 21 and plugging it in Equation 20).

Thanks for the comment. We agree with the reviewer on the point of the fast slow approximation. However, while responding to a point raised by Reviewer 1 (Point P 1.5), we realized that while the characteristic function gives good intuitions about why SFA cannot stabilize the system, the particular approximation is not fully justified. To address this point, we therefore analyzed the full 3D system instead and, consequently, the corresponding analysis has changed substantially in the revised manuscript. Specifically, we now provide the asymptotic stability analysis for the three-dimensional dynamical system for the cases of SFA, E-to-E STD, and E-to-I STF in the new method sections with the titles:

– Stability conditions in networks with SFA (Methods)

– Stability conditions in networks with E-to-E STD (Methods)

– Stability conditions in networks with E-to-I STF (Appendix 1)

In the case of STP, the reduction to the characteristic function F(z) is still valid and illuminating, because a stable high-activity fixed point exists. Due to a separation of timescales between the STP variable and the rate dynamics, we can approximate with its fixed point value ≈^∗ or the corresponding previous fixed value in the case of input changes (Figure 2). This allows us to visualize the change of network stability via the characteristic function. This is now stated clearly in a dedicated Methods section with the title “Characteristic function approximation for networks with E-to-E STD.”

6 – Switching to ISN with stimulus from a non-ISN baseline is presumably parameter dependent, and this dependence should be characterized better (at least give examples of networks in which this is not the case, in a supplementary figure, and correspondingly weaken the absolute statement in lines 192-194 so that it says this transition "can" happen, which would avoid implying that this is necessarily the case, assuming it's not).

Thanks for raising this valid point. Indeed, the switch from ISN to non-ISN is parameter dependent and in the revised manuscript we now state it as such and characterize this transition extensively. First, we edited and added the following sentences making the point to the relevant part of the Results section:

“We found that in networks with STP the ISN index can switch sign from negative to positive during external stimulation, indicating that the ensemble can transition from a non-ISN to an ISN (Figure 2F). Notably, this behavior is distinct from linear network models in which the network operating regime is independent of the input (Methods). Whether this switch between non-ISN to ISN occurred, however, was parameter dependent and we also found network configurations that were already in the ISN regime at baseline and remained ISNs during stimulation (Figure 2—figure supplement 5). Thus, re-stabilization was largely unaffected by the network state and consistent with experimentally observed ISN states.”

To make the conditions more concise, we added the mathematical definition of ISN index in networks with STP to the new Methods section “Conditions for ISN in networks with E-to-E STD” and the new Appendix section “Conditions for ISN in networks with E-to-I STF” and added Figure 2—figure supplement 5 showing an example network which is already in ISN at baseline. Finally, Figure 2—figure supplement 7A illustrates by plotting the ISN index as a function of input_E that the inhibition stabilization property of the network depends on the input strength_E.

Also in the ISN index (Equation 13) shouldn't J_EE be multiplied by x (depression variable) at the fixed point?

This is now fixed. We added the Method section “Conditions for ISN in networks with E-to-E STD” and the variable appears in the relevant expression for the ISN index (Equation (56)).

7 – I found the part about pattern separation (p. 14) not clearly written and motivated. In fact I think the term "pattern separation" should be replaced with "stimulus selectivity". Pattern separation is the appropriate term when a networks output patterns are more separable than the patterns at the input-level. However, in the current case in which the external input is given only to E1 (only to one of the two E populations), the input is fully separable, so the network is potentially only capable of reducing separation. At any rate the pattern separation would be meaningful if they had compared separation at input-level (distance from decision boundary in gE1 – gE2 plane) with separation at output-level (distance from decision boundary in rE1 – rE2 plane). I think what is currently shown and charcterized is stimulus selectivity and the result is that transient onset responses are more selective than fixed point responses. So this section (and corresponding parts in intro/discussion) should be rewritten to convey this.

We agree with the reviewer that the setup in Figure 4 really addresses questions pertaining to pattern selectivity rather than pattern separation, which is rather the topic of Figure 5.

To address this point, we changed the figure and text in the section “NTA provides better pattern completion than fixed points while retaining stimulus selectivity” to refer to selectivity rather than pattern separation and we discuss pattern separation in the next section “NTA provides higher amplification and pattern separation in morphing experiments.”

Specifically, we expanded the relevant paragraph at the end of this section. It now reads:

“Further, we examined the impact of competition through lateral inhibition as a function of the E-to-I inter-ensemble strength J’_IE (Methods). As above, we quantified its impact by measuring the representational distance to the decision boundary for the transient onset responses and fixed point activity. We found that regardless of the specific STP mechanism, the distance was larger for the onset responses than for the fixed point activity, consistent with the notion that the onset dynamics separate stimulus identity reliably (Figure 5D, E). Since the absolute activity levels between onset and fixed point differed substantially, we further computed the relative pattern Separation index (r_E₂ − r_E₁)∕(r_E₁ + r_E₂) and found that the onset transient provides better pattern separation ability for ambiguous stimuli with close to 0.5 (Figure 5—figure supplement 1) provided that the E-to-I connection strength across ensembles J’_IE^′ is strong enough. All the while separability for the onset transient was slightly decreased for distinct inputs with ∈ {0,1} in comparison to the fixed point. In contrast, fixed points clearly separated such pure stimuli while providing weaker pattern separation for ambiguous input combinations. Importantly, these findings qualitatively held for networks with NTA mediated by E-to-I STF (Figure 5—figure supplement 2). Thus, NTA provides stronger amplification and pattern separation than fixed point activity in response to ambiguous stimuli.”

7a – On the other hand, this finding (stronger selectivity at onset, vs steady-state or sustained response) is presented as a desired outcome. This may be so, but it seems inconsistent with empirical findings on orientation selectivity in (mouse) visual cortex in which orientation selectivity is much stronger for sustained responses compared to onset transients. This should be mentioned (e.g. in lines 290, 326, and 418-420).

Thanks for raising this interesting point. Unfortunately, we were unable to find a clear line of literature arguing either for or against this statement. On the one hand, we found sources that might suggest an increase of selectivity at steady state (Ringach et al., 1997; Miller et al., 2001), while other studies (in primates) seem to suggest the opposite (Celebrini et al., 1993). Importantly, the comparison is hampered, because in many studies the input stimulus does not seem to be on for long enough to allow for the system to settle in a steady state (Ringach et al., 1997). Finally, the fact we did not quantify the width of tuning curves of continuous stimuli in the present study further complicates the direct comparisons between our model and the above studies.

To, nevertheless, address this point, we rephrased our conclusions more carefully and added the following sentence to the Discussion (formerly 418-420):

“Yet, it remains to be seen whether these findings are also coherent with data on the temporal evolution in other sensory systems.”

7b – The distance to decision boundary used to characterize "pattern separation" is not clearly described in lines 280-282. The description in the Methods section was also somewhat ambiguous, but giving a formula there would solve this issue. What I understood is that the measure is basically rE1 – rE2 where rE1 and rE2 are population mean activities of the E neurons in ensembles 1 and 2. If so, why not use the normalized measure (rE1 – rE2)/(rE1 + rE2)?

Thanks for making this point. We used an absolute distance measure to capture that the difference in absolute firing rate, which is what a downstream neuron can directly read out, is much larger during the onset transient. In contrast, the relative distance measure always goes to ±1 if one of the populations has zero activity, which happens in our example at the fixed point at which the overall activity might still be very low and hence hard to read out by downstream brain areas. Thus, pattern selectivity at the fixed point is high, but it is not clear whether it could be read-out if the overall activity is too low. We now add both the absolute distance measure and a relative measure in our analysis of the morphing experiments (see below).

In any case, we now provide the formula for the distance to the decision boundary in the Methods section “Distance to the decision boundary.” Moreover, we added an inset to Figure 4A which illustrates this measure.

8 – The property B is depicted in various places (e.g. line 410) by saying NTA requires or relies on symmetric E-E connections. While NTA clearly relies on the destabilizing effect of recurrent EE connections, it is not clear that the symmetry (at least exact symmetry) of these connections is necessary. Indeed their LIF spiking net has random connectivity, so the EE part of connectivity at the level of single neurons in that network is actually not symmetric. These sentences could thus be more accurately written to clarify this point.

We agree, this has now been amended. We extensively revised our Abstract, Introduction and Discussion and now speak of recurrent excitation rather than (exact) symmetry. Please see our response to Points P 1.3 and P 2.2.

9 – In the current study, external stimulus was only given to excitatory neurons, with authors saying the study of feedforward input to inhibition is beyond the scope of this work, yet they also say (lines 429-432) that "we are confident that our main findings remain unaffected in the presence of substantial feedforward inhibition", and cite supplementary figure 1, which however was for the case without STP. I think it would be easy to provide a supplementary figure characterizing NTA in a network with STP in which gI is nonzero for stimulus (e.g. gI = gE).

We fully agree. We added Figure 2—figure supplement 12 to address this point. The figure characterizes the amplification in networks with E-to-E STD in the presence of additional feedforward inhibitory inputs. It shows that NTA can still emerge even when stimulus causes a change in feedforward inhibition_I that is by far greater than the change in feedforward excitation_E.

In the Discussion we now refer to this figure as follows:

“While an in-depth comparison for different origins of inhibition was beyond the scope of the present study, we found that increasing the inputs to the excitatory population and inhibitory population by the same amount can still lead to NTA (Figure 1—figure supplement 1; Figure 2—figure supplement 12; Methods), suggesting that our main findings can remain unaffected in the presence of substantial feedforward inhibition.”

References

Ahmadian Y, Rubin DB, Miller KD. Analysis of the stabilized supralinear network. Neural Computation 2013;25(8):1994–2037. doi: https://doi.org/10.1162/NECO_a_00472.

Amit DJ, Brunel N. Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex. Cerebral Cortex 1997;7(3):237–252. doi: https://doi.org/10.1093/cercor/7.3.237.

Baker C, Zhu V, Rosenbaum R. Nonlinear stimulus representations in neural circuits with approximate excitatoryinhibitory balance. PLoS Computational Biology 2020;16(9):1–30. doi: https://doi.org/10.1371/journal.pcbi. 1008192.

Bolding KA, Nagappan S, Han BX, Wang F, Franks KM. Recurrent circuitry is required to stabilize piriform cortex odor representations across brain states. eLife 2020;9:1–23. doi: https://doi.org/10.7554/eLife.53125.

Bondanelli G, Ostojic S. Coding with transient trajectories in recurrent neural networks. PLoS Computational Biology 2020;16(2):1–36. doi: https://doi.org/10.1371/journal.pcbi.1007655.

Carrillo-Reid L, Yang W, Bando Y, Peterka DS, Yuste R. Imprinting and recalling cortical ensembles. Science 2016;353(6300):691–694. doi: https://doi.org/10.1126/science.aaf7560.

Cayco-Gajic NA, Silver RA. Re-evaluating Circuit Mechanisms Underlying Pattern Separation. Neuron 2019;101(4):584–602. doi: https://doi.org/10.1016/j.neuron.2019.01.044.

Celebrini S, Thorpe S, Trotter Y, Imbert M. Dynamics of orientation coding in area VI of the awake primate. Visual Neuroscience 1993;10(5):811–825. doi: https://doi.org/10.1017/S0952523800006052.

Christodoulou G, Vogels TP, Agnes EJ. Regimes and mechanisms of transient amplification in abstract and biological networks. bioRxiv 2021; doi: https://doi.org/10.1101/2021.04.01.437964.

Compte A, Brunel N, Goldman-Rakic PS, Wang XJ. Synaptic mechanisms and network dynamics underlying spatial working memory in a cortical network model. Cerebral Cortex 2000;10(9):910–923. doi: https://doi.org/10.1093/ cercor/10.9.910.

DeWeese MR, Wehr M, Zador AM. Binary spiking in auditory cortex. Journal of Neuroscience 2003;23(21):7940– 7949. doi: https://doi.org/10.1523/JNEUROSCI.23-21-07940.2003.

Echeveste R, Aitchison L, Hennequin G, Lengyel M. Cortical-like dynamics in recurrent circuits optimized for sampling-based probabilistic inference. Nature Neuroscience 2020;23(9):1138–1149. doi: http://dx.doi.org/10.1038/s41593-020-0671-1.

Gillett M, Pereira U, Brunel N. Characteristics of sequential activity in networks with temporally asymmetric Hebbian learning. Proceedings of the National Academy of Sciences of the United States of America 2020;117(47):29948–29958. doi: https://doi.org/10.1073/pnas.1918674117.

Goldman MS. Memory without feedback in a neural network. Neuron 2009;61(4):621–634. doi: https://doi.org/10. 1016/j.neuron.2008.12.012.

Hennequin G, Ahmadian Y, Rubin DB, Lengyel M, Miller KD. The dynamical regime of sensory cortex: Stable dynamics around a single stimulus-tuned attractor account for patterns of noise variability. Neuron 2018;98(4):846– 860.e5. doi: https://doi.org/10.1016/j.neuron.2018.04.017.

Hennequin G, Vogels TP, Gerstner W. Non-normal amplification in random balanced neuronal networks. Physical Review E – Statistical, Nonlinear, and Soft Matter Physics 2012;86(1):1–12. doi: https://doi.org/10.1103/PhysRevE.86.011909.

Hennequin G, Vogels TP, Gerstner W. Optimal control of transient dynamics in balanced networks supports generation of complex movements. Neuron 2014;82(6):1394–1406. doi: https://doi.org/10.1016/j.neuron.2014.04.045.

Kraynyukova N, Tchumatchenko T. Stabilized supralinear network can give rise to bistable, oscillatory, and persistent activity. Proceedings of the National Academy of Sciences of the United States of America 2018;115(13):3464–3469. doi: https://doi.org/10.1073/pnas.1700080115.

Levina A, Herrmann JM, Geisel T. Dynamical synapses causing self-organized criticality in neural networks. Nature Physics 2007;3(12):857–860. doi: https://doi.org/10.1038/nphys758.

Litwin-Kumar A, Doiron B. Slow dynamics and high variability in balanced cortical networks with clustered connections. Nature Neuroscience 2012;15(11):1498–1505. doi: https://doi.org/10.1038/nn.3220.

Loebel A, Nelken I, Tsodyks M. Processing of sounds by population spikes in a model of primary auditory cortex. Frontiers in Neuroscience 2007;1:15. doi: https://doi.org/10.3389/neuro.01.1.1.015.2007.

Loebel A, Tsodyks M. Computation by Ensemble Synchronization in Recurrent Networks with Synaptic Depression. J Comput Neurosci 2002 Sep;13(2):111–124. doi: https://doi.org/10.1023/A:1020110223441.

Markram H, Wang Y, Tsodyks M. Differential signaling via the same axon of neocortical pyramidal neurons. Proceedings of the National Academy of Sciences of the United States of America 1998;95(9):5323–5328. doi: https://doi.org/10.1073/pnas.95.9.5323.

Marshel JH, Kim YS, Machado TA, Quirin S, Benson B, Kadmon J, et al. Cortical layer-specific critical dynamics triggering perception. Science 2019;365(6453). doi: https://doi.org/10.1126/science.aaw5202.

Mazzucato L, La Camera G, Fontanini A. Expectation-induced modulation of metastable activity underlies faster coding of sensory stimuli. Nature Neuroscience 2019;22(5):787–796. doi: https://doi.org/10.1038/ s41593-019-0364-9.

Miller KD, Pinto DJ, Simons DJ. Processing in layer 4 of the neocortical circuit: New insights from visual and somatosensory cortex. Current Opinion in Neurobiology 2001;11(4):488–497. doi: https://doi.org/10.1016/ S0959-4388(00)00239-7.

Murphy BK, Miller KD. Balanced amplification: A new mechanism of selective amplification of neural activity patterns. Neuron 2009;61(4):635–648. doi: https://doi.org/10.1016/j.neuron.2009.02.005.

Ozeki H, Finn IM, Schaffer ES, Miller KD, Ferster D. Inhibitory Stabilization of the Cortical Network Underlies Visual Surround Suppression. Neuron 2009;62(4):578–592. doi: http://doi.org/10.1016/j.neuron.2009.03.028.

Pala A, Petersen CCH. in vivo measurement of cell-type-specific synaptic connectivity and synaptic transmission in layer 2/3 mouse barrel cortex. Neuron 2015;85(1):68–75. doi: https://doi.org/10.1016/j.neuron.2014.11.025.

Plenz D, Thiagarajan TC. The organizing principles of neuronal avalanches: cell assemblies in the cortex? Trends in Neurosciences 2007 Mar;30(3):101–110. doi: https://doi.org/10.1016/j.tins.2007.01.005.

Ponce-Alvarez A, Thiele A, Albright TD, Stoner GR, Deco G. Stimulus-dependent variability and noise correlations in cortical MT neurons. Proceedings of the National Academy of Sciences of the United States of America 2013;110(32):13162–13167. doi: https://doi.org/10.1073/pnas.1300098110.

Ringach DL, Hawken MJ, Shapley R. The dynamics of orientation tuning in macaque primary visual cortex. Investigative Ophthalmology and Visual Science 1997;38(4):281–284. doi: https://doi.org/10.1038/387281a0.

Rubin DB, VanHooser SD, Miller KD. The stabilized supralinear network: A unifying circuit motif underlying multiinput integration in sensory cortex. Neuron 2015;85(2):402–417. doi: https://doi.org/10.1016/j.neuron.2014.12. 026.

Rupprecht P, Friedrich RW. Precise synaptic balance in the zebrafish homolog of olfactory cortex. Neuron 2018;100(3):669–683.e5. doi: https://doi.org/10.1016/j.neuron.2018.09.013.

Sadeh S, Clopath C. Inhibitory stabilization and cortical computation. Nature Reviews Neuroscience 2021;22(1):21–37. doi: http://doi.org/10.1038/s41583-020-00390-z.

Sanzeni A, Akitake B, Goldbach HC, Leedy CE, Brunel N, Histed MH. Inhibition stabilization is a widespread property of cortical networks. eLife 2020;9:1–39. doi: https://doi.org/10.7554/eLife.54875.

Stern M, Bolding KA, Abbott LF, Franks KM. A transformation from temporal to ensemble coding in a model of piriform cortex. eLife 2018;7:1–26. doi: https://doi.org/10.7554/eLife.34831.

Tsodyks MV, Markram H. The neural code between neocortical pyramidal neurons depends on neurotransmitter release probability. Proceedings of the National Academy of Sciences of the United States of America 1997;94(2):719–723. doi: https://doi.org/10.1073/pnas.94.2.719.

Tsodyks MV, Skaggs WE, Sejnowski TJ, McNaughton BL. Paradoxical effects of external modulation of inhibitory interneurons. Journal of Neuroscience 1997;17(11):4382–4388. doi: https://doi.org/10.1523/jneurosci.17-11-04382.1997.

Van Vreeswijk C, Hansel D. Patterns of synchrony in neural networks with spike adaptation. Neural Computation 2001;13(5):959–992. doi: https://doi.org/10.1162/08997660151134280.

Van Vreeswijk C, Sompolinsky H. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science 1996;274(5293):1724–1726. doi: https://doi.org/10.1126/science.274.5293.1724.

Vinje WE, Gallant JL. Sparse coding and decorrelation in primary visual cortex during natural vision. Science 2000;287(5456):1273–1276. doi: http://doi.org/10.1126/science.287.5456.1273.

Zucker RS, Regehr WG. Short-term synaptic plasticity. Annual Review of Physiology 2002;64:355–405. doi: https://doi.org/10.1146/annurev.physiol.64.092501.114547.

https://doi.org/10.7554/eLife.71263.sa2

Article and author information

Author details

Yue Kris Wu
1. Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland
2. Faculty of Natural Sciences, University of Basel, Basel, Switzerland
3. Max Planck Institute for Brain Research, Frankfurt, Germany
4. School of Life Sciences, Technical University of Munich, Freising, Germany
Contribution
Formal analysis, Investigation, Methodology, Software, Visualization, Writing – original draft, Writing – review and editing

Competing interests
No competing interests declared

"This ORCID iD identifies the author of this article:" 0000-0002-9804-2537
Friedemann Zenke
1. Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland
2. Faculty of Natural Sciences, University of Basel, Basel, Switzerland
Contribution
Conceptualization, Funding acquisition, Methodology, Supervision, Writing – original draft, Writing – review and editing

For correspondence
friedemann.zenke@fmi.ch

Competing interests
No competing interests declared

"This ORCID iD identifies the author of this article:" 0000-0003-1883-644X

Funding

Novartis Foundation

Yue Kris Wu
Friedemann Zenke

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

We thank Rainer W Friedrich, Claire Meissner-Bernard, William F Podlaski, and members of the Zenke Group for comments and discussions. This work was supported by the Novartis Research Foundation.

Senior Editor

Ronald L Calabrese, Emory University, United States

Reviewing Editor

Timothy O'Leary, University of Cambridge, United Kingdom

Publication history

Preprint posted: June 10, 2021 (view preprint)
Received: June 14, 2021
Accepted: December 10, 2021
Accepted Manuscript published: December 13, 2021 (version 1)
Version of Record published: February 7, 2022 (version 2)

Copyright

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.