Many, if not most, of the tasks the brain performs are inherently temporal in nature: from recognizing and generating complex spatiotemporal patterns—such as a phoneme sequence that composes a word—to creating temporal expectations of when an event will occur (Mauk and Buonomano, 2004; Nobre et al., 2007; Ivry and Schlerf, 2008; Merchant et al., 2013; Hopfield, 2015). In contrast to the representation of an object in a static image, such as a picture of a face, robust decoding and encoding of a time-varying patterns must rely on the spatiotemporal dependencies inherent to the pattern. How does the brain accomplish this? One highly influential theory in neuroscience holds that information is stored as fixed-point attractors that emerge in the dynamic activity of the brain’s recurrently connected circuits (Hopfield, 1982; Hopfield and Tank, 1986; Amit and Brunel, 1997; Wang, 2001). Two limitations of this framework are: (1) It fails to capture the temporal aspect of stored information, thus forcing many computational models to ‘spatialize’ time—that is, they treat the temporal component of time-varying patterns as additional spatial dimensions (Rabiner, 1989; Waibel et al., 1989; Elman, 1990; Hinton et al., 2012; Mnih et al., 2015); (2) it does not capture a fundamental feature of how the brain processes temporal information: temporal invariance. For example, humans readily recognize temporally warped—compressed or dilated—speech or music.

While the mechanisms that underlie the brain’s ability to perform a broad range of spatiotemporal tasks in the sensory and motor domains are not known, there is mounting theoretical and experimental evidence that our ability to tell time on the sub-second scale, and represent time-varying patterns, relies on the inherent continuous dynamics, and computational potential, of recurrent neural networks (Rabinovich et al., 2008; Buonomano and Maass, 2009; Mante et al., 2013; Crowe et al., 2014; Carnevale et al., 2015; Li et al., 2016). This framework alleviates the restrictions imposed by traditional discrete-time or fixed-point attractor models, by relying on the recurrent connections to implicitly maintain an ongoing memory of the pattern. Indeed, recent computational studies have established that by tuning the weights within recurrent neural network (RNN) models, it is possible to robustly store and generate complex time-varying motor patterns (Laje and Buonomano, 2013; Abbott et al., 2016; Rajan et al., 2016). What remains unknown is whether the same approach can be used to robustly discriminate time-varying sensory patterns. Indeed, this poses a challenging problem because in order to effectively process spatiotemporal stimuli the dynamics of an RNN must be sensitive to the relevant spatial and temporal features of the sensory stimuli, while being able to generalize across their natural spatial and temporal variations.

Neuroscientists have typically distinguished between sensory and motor areas; but it is well established that activity in sensory areas is strongly influenced by motor behavior, and that sensory stimuli can modulate activity in motor areas—furthermore, some brain areas are characterized as being sensorimotor (Doupe and Kuhl, 1999; Ayaz et al., 2013; Chang et al., 2013; Schneider and Mooney, 2015; Cheung et al., 2016). Computationally, sensory and motor processing are understood to have disparate requirements—during sensory processing, network dynamics should primarily be driven by the sensory inputs; in contrast, during motor processing neural dynamics should be autonomous and driven primarily by recurrent interactions. It remains unclear how a single network could accomplish both tasks, and couple them when necessary. Indeed, to date, no previous models have shown that the same network can satisfy the requirements for both sensory and motor processing. Here we show that the same RNN can reliably function in both a sensory and motor regime. Specifically the same RNN can convert complex time-varying sensory patterns into motor patterns—thus performing a transcription task in which spoken digits are identified and read out as ‘handwritten’ digits.

In the temporal domain, understanding how the brain recognizes temporally compressed or dilated patterns represents a long-standing challenge. This temporal invariance is particularly evident in our ability to recognize temporally warped speech or music (Miller et al., 1984; Sebastián-Gallés et al., 2000). Our results suggest that one advantage of storing time-varying patterns as neural trajectories within RNNs is that, under the appropriate conditions, this strategy naturally accounts for temporal invariance.

We first asked if a single RNN could perform a complex sensory-motor task: transcribing spoken digits into handwritten digits. A continuous-time firing-rate RNN with randomly assigned sparse recurrent connections was used (Sompolinsky et al., 1988). The strengths of these recurrent connections were initialized to be relatively strong; thus, before training the network was in a so-called high-gain regime (g = 1.6) that is characterized by self-perpetuating and chaotic activity (Materials and Methods). The transcription task is divided into a sensory and a motor epoch. During the sensory epoch, the RNN is presented with a spoken digit, and over the ensuing motor epoch the RNN drives an output pattern transcribing the presented digit via three motor outputs x, y, and z—activity in the x and y units determines the 2D coordinates of a ‘pen on paper’, while z determines if the ‘pen’ is in contact with the ‘paper’ or not. Figure 1A illustrates the network architecture and transcription task for the digit ‘2’. Successful performance of this transcription task requires that the RNN: 1) encode spoken digits with a set of neural trajectories in high-dimensional phase space; 2) autonomously generate digit-specific trajectories during the motor epoch, in order to drive the digit-specific output patterns; and most importantly 3) generate trajectories that are stable so they may encode each digit’s sensory pattern, sensory-motor transition, and motor pattern in a manner that is invariant not just to background noise but also to spatiotemporal variations of the spoken digits, including temporal warping.

Figure 1 with 1 supplement see all

Download asset Open asset

Video 1

Download asset

We attempted to satisfy these three conditions by training the RNN using a supervised learning rule, in which the recurrent units were trained to robustly reproduce their ‘innate’ patterns of activity (Laje and Buonomano, 2013)—that is, those generated in the untrained network—using the recursive-least-squares learning rule (Haykin, 2002; Sussillo and Abbott, 2009). In essence, the RNN is trained to reproduce one digit-specific ‘innate’ trajectory in response to all training utterances of a given digit. For example, the pattern of activity produced in response to a ‘template’ utterance of a given digit from the beginning of the sensory epoch to the end of the motor epoch is taken as the innate trajectory; the network is then trained to reproduce this trajectory in response to other utterances of the same digit. Furthermore, it is trained to do so regardless of the initial state of the network’s units, and in the presence of continuous background noise (Materials and Methods). Only after training the RNN (recurrent training), are the output units trained to generate the handwriting patterns representing each of the digits during the motor epoch (output training)—this separation of the recurrent and output training phases, while not necessary, allows for rapid training and retraining of the outputs to produce arbitrary motor patterns, without retraining the recurrent weights. Notably, the temporal separation of sensory and motor learning is observed in some forms of sensori-motor learning (Doupe and Kuhl, 1999). The output training phase is performed using standard supervised methods (Materials and Methods). Figure 1B shows the outputs of a trained network cross-tested on ten digits (0–9) across five speakers. Following training, the network successfully transcribes the utterances used during training (marked by asterisks). More importantly, performance generalizes to novel utterances and speakers in the dataset despite significant variations in the duration and spatiotemporal structure across utterances (Video 1).

To quantify performance, we need an objective measure of the quality of the motor output for each test utterance. This itself represents a classification task, wherein images of RNN-generated transcriptions must be assigned to one of the ten possible digits. Rather than use a human-based performance measure, we used a standard deep convolutional neural network (CNN) (LeCun et al., 2015) to rate the performance of trained RNNs (Materials and Methods)—i.e., the CNN was used to determine if each ‘handwritten’ digit was correct or not. Performance of trained RNNs was 98.7% on a test set of 410 novel stimuli (Figure 1C). Since untrained RNNs (‘reservoir networks’) are in and of themselves capable of performing many interesting computations (Maass et al., 2002; Jaeger and Haas, 2004; Lukoševičius and Jaeger, 2009), it is important to determine how much of this performance is dependent on the training of the RNNs. We thus examined a control group wherein the three output units are trained as they were for the trained RNNs, but the RNN itself was not trained (the ‘reservoir’). The performance was very poor (20%), in large part because during the motor epoch a reservoir RNN is operating in an autonomous mode that is chaotic—making it difficult for the output units to learn to produce the target output patterns. We also examined the effects of training an RNN only during the motor epoch, resulting in a performance of 71%. This significant improvement over the reservoir networks is a consequence of the fact that a network driven by external inputs can encode sensory stimuli despite a lack of sensory epoch training, due to the intrinsic ability of RNNs to encode sensory stimuli and the stabilizing influence of the external inputs (see below). Yet, despite the improvement in performance, some digits were almost always misclassified (Figure 1—figure supplement 1). The difference in performance between the trained RNNs (sensory and motor training) and the exclusively motor trained RNNs, confirms the importance of tuning the recurrent weights to the sensory discrimination component of the task (see below).

Stability of the neural trajectories

RNNs operating in high-gain regimes are prone to chaotic behavior (Sompolinsky et al., 1988; Rajan et al., 2010a) because the strong recurrent feedback of these networks rapidly amplifies any noise or perturbations. It is thus critical to demonstrate that the above performance is robust to background noise and perturbations during the sensory and motor epochs. The distinction between epochs is critical since the RNN is operating in fundamentally different regimes during the sensory and motor epochs. During the sensory epoch the recurrently generated internal dynamics are partially suppressed (or ‘clamped’) as a result of the external input, yet during the motor epoch all activity is internally generated. To examine the stability of these sensory and motor ‘object’ representations, we briefly perturbed the internal dynamics of the RNN during either the sensory or motor epochs. Sensory epoch perturbations were introduced half way through the presentation of each utterance, while motor epoch perturbations were introduced at the 10% mark of the motor epoch. Figure 2A shows the activity of a hundred units from a trained RNN (and the resulting output), when it is presented with the digit ‘three’ and strongly perturbed (amplitude = 2) in the motor epoch. The resulting transcription briefly deviates from an unperturbed one (grey backdrop), but recovers mid-voyage. The sensory epoch is less sensitive to perturbations—as would be expected because the presence of the external input serves as a stabilizing influence (Rajan et al., 2010a). During the motor epoch the RNN is operating autonomously as a ‘dynamic attractor’, that is, once bumped off its trajectory it maintains a memory of its current voyage and is able to return to the original trajectory (Laje and Buonomano, 2013) (Figure 2B). Performance measurements using the CNN classifier reveal a graceful degradation across a wide range of perturbation magnitudes, and confirm the superior robustness of the sensory epoch (Figure 2C).

Figure 2

Download asset Open asset

RNN training sculpts network dynamics to enhance discrimination

How does tuning the recurrent weights allow the RNN to stably encode both sensory and motor information in neural trajectories, despite considerable spatiotemporal differences between digit utterances (Figure 1B, insets)? Firing rate traces of sample units in a trained RNN show more similar patterns of activity in response to different utterances of the same digit, when compared to a reservoir RNN (Figure 3A–B). To better examine the structure of the population representations, we can visualize and compare the neural trajectories in response to multiple utterances (learned and novel) of the same digit, during the sensory and motor epochs, in principal component analysis (PCA) subspace. The trajectories produced in the untrained network by utterances of the digits ‘six’ and ‘eight’, during the sensory epoch, occupied a fairly large abutting volume of PCA subspace (Figure 3C); and during the motor epoch the trajectories were highly variable—the network strongly and perpetually amplifies the differences between utterances, because it is in a chaotic regime. In contrast, in the trained RNN, sensory epoch trajectories of each digit were restricted to a narrower volume, and better separated from the volume representing the other digit (Figure 3D). During the motor epoch, the trajectories for different utterances of the same digit were constrained to a much narrower tube—reflecting the dynamic attractor—and better separated from the motor trajectories of the other digit. The set of all within-digit trajectories can be thought of as populating a hypertube that delimits the volume of phase space traversed by the digit. During the sensory epoch, this hypertube of trajectories represents a memory of a family of related spatiotemporal objects: the spoken digit. During the motor epoch the network is autonomous, and the hypertube of all within-digit trajectories narrows to a dynamic attractor that represents the ‘motor memory’ of the corresponding handwritten digit.

Figure 3

Download asset Open asset

It is well established that cortical circuits undergo experience-dependent plasticity—a process that seems to result in the optimization or specialization of those circuits to the tasks the animal is exposed to (Buonomano and Merzenich, 1998; Crist et al., 2001; Feldman and Brecht, 2005; Karmarkar and Dan, 2006). The above results establish that training the RNN does improve discrimination performance—the recurrent weights are tuned to the task at hand—but leaves open the question of how exactly this is accomplished. To answer this question, we asked if the Euclidean distances between trajectories in response to different utterances of the same digit (within-digit distance), and utterances of different digits (between-digit distance), were significantly altered in comparison to the reservoir (untrained) RNN. Training significantly decreases the mean within-digit distances during the sensory epoch (Figure 4A). Importantly, in doing so, it does not diminish the large separations between trajectories of different digits. The same effect, much enhanced, is observed during the motor epoch, a consequence of the formation of dynamic attractors. Successful formation of these attractors is strongly influenced by an unambiguous and stable sensory epoch-motor epoch transition of the network dynamics. Such a transition relies on two factors: (i) a low within-digit separation at the end of the sensory epoch trajectory, ensuring initial conditions at the beginning of the motor epoch that lie within the initial basin of attraction of the motor pattern-encoding dynamic attractor; (ii) a between-digit separation that is substantially larger than the within-digit separation of sensory epoch trajectories, particularly at the end of the sensory epoch, to ensure robust and divergent sensory-epoch-motor epoch transitions for different digits.

Figure 4

Download asset Open asset

The finding that training sculpts the dynamics of the RNN during the sensory epoch is also a critical one because it shows that even though RNN activity is governed in part by the external input, the internal connections are critical: tuning them effectively improves the interaction between the sensory input and the internally generated dynamics. This improvement is expressed as a collapsing of the family of trajectories representing a digit into a narrower hypertube. These results follow as a direct consequence of the supervised recurrent training paradigm (Materials and Methods)—a single innate trajectory serves as a common target for different training utterances of a digit, thereby inducing changes in the network’s recurrent dynamics that are necessary to encode the different utterances along similar neural trajectories; However, different digits are encoded in rapidly divergent trajectories because the target trajectories for each digit are generated by an untrained chaotic network. Further evidence that training dramatically influences the weight matrix of the RNN is demonstrated by the change in its eigenspectrum (Figure 4B). Because the network is initialized with normally distributed weights with a gain of 1.6 (Materials and Methods), the eigenspectrum of the reservoir’s weight matrix lies in a circle of radius approximately 1.6. Training results in a compression of those eigenvalues with real part larger than one, bringing the maximal real part of the eigenvalues closer to one. In a linear network, when all eigenvalues have a real part less than one, it implies that the network’s activity will decay to zero when it operates autonomously; however our network is nonlinear and does not operate exclusively in the autonomous mode, so interpretations of the eigenspectrum of the weight matrix are not straightforward—nevertheless the compression of the eigenspectrum is consistent with the increased stability of the network during the sensory and motor epochs (Rajan and Abbott, 2006; Ostojic, 2014).

Balance between recurrent and input dynamics is crucial for discrimination

The above results provide insights as to how a single neural circuit can function in two seemingly distinct computational modes: sensory and motor. Sensory discrimination requires circuits to be highly responsive to external stimuli in order to categorize inputs into discrete classes. In contrast, motor tasks require autonomous generation of spatiotemporal patterns, and thus need the network dynamics to be somewhat resistant to external inputs. In the network described here, this balance depends on the recurrent weights and the magnitude of the external drive.

The recurrent weights must be strong—that is the network must be in a high-gain regime—for two reasons. First, network dynamics in high-gain regimes are high-dimensional (Rajan et al., 2010b), which naturally yields well-separated trajectories for different digits. A goal of training then, is to achieve recurrent suppression of within-digit separation, while retaining the large between-digit separation (Figure 4). Second, as noted earlier, a network operating in a high-gain regime is inherently capable of generating the self-perpetuating activity required in the motor epoch, which is then stabilized by the training procedure. Thus during the sensory epoch, the RNN is operating in a regime that is both sensitive to external input and influenced by strong internal recurrent dynamics. In the motor mode, the RNN operates autonomously, generating locally stable trajectories that serve as a high-dimensional ‘engine’ that drives arbitrary low dimensional output patterns.

The ability of the network to meaningfully process input patterns during the sensory epoch, so that they are easy to discriminate, also depends on the strength of the inputs. To better understand the impact of input amplitude on the encoding trajectories and their discriminability, before and after training, we parametrically varied the input amplitude and studied the resulting RNN dynamics during the sensory epoch. In the reservoir, both within and between-digit distances diminish as the input amplitude increases (Figure 5A, left panel), consistent with a previous study showing that strong inputs can progressively dominate and override the internal dynamics of an RNN (Rajan et al., 2010a). In the extreme, input-dominated regimes effectively void the recurrent weights and render training useless (Figure 5A, right panel). The poor discriminability at high input amplitudes is further confirmed by dimensionality measurements of sensory epoch trajectories (Materials and Methods), which mirror the low-dimensionality of the input cochleograms, regardless of training (Figure 5B). In contrast, low-input amplitude regimes are dominated by the RNN’s internal dynamics. Reservoirs in this regime produce chaotic high-dimensional dynamics and are insufficiently sensitivity to external inputs; thus the within- and between-digit distances are similarly high (Figure 5A, left). Training strongly alters these dynamics, lowering its dimensionality (Figure 5B). However, as a consequence of poor input-sensitivity, these changes fail to improve discrimination—trained RNNs in this regime still sustain similar within and between-digit distances (Figure 5B, right). RNNs trained at intermediate input amplitudes (0.5, 5) are both sensitive to the input and able to discriminate between digits, because their sensory epoch trajectories are shaped both by the input and internal dynamics. It is thus critical that the input drive be strong enough to influence ongoing activity but not strong enough to ‘erase’ recent information encoded in the current trajectory.

Figure 5

Download asset Open asset

Mechanisms underlying spatial (Spectral) Generalization

The attracting dynamics of the network result in each digit being represented in a narrow hypertube. These representations must be robust across both spectral and temporal (changes in speed) differences in the utterances of each digit. Next, we separately examine the mechanisms underlying spatial and temporal generalization. Variation in the spectral structure of digits—spectral noise (Figure 6A)—is qualitatively different from background noise. First, spectral noise exhibits time-dependent structured relationships with the input signal (they may be correlated, anti-correlated or tend to occur along specific directions during specific parts of the input). For example, the spectral differences between utterances of one phoneme within a digit will be different than those of another. Invariance then requires the network to isolate and integrate the signal while suppressing signal-dependent spectral noise. Second, spectral noise is a form of correlated noise. The noise in each input channel is simultaneously delivered to multiple recurrent units via common input projections. Moreover, the spectral noise in different channels, particularly neighboring ones, exhibit structured time-dependent relationships. Third, spectral noise is proportional in magnitude to the input amplitude, and therefore tends to be much stronger than the background noise that is typically injected during training and testing.

Figure 6 with 1 supplement see all

Download asset Open asset

To better characterize how spectral noise invariance emerges during training, we measured network responses while naturally and artificially varying the structure of the spectral noise. Any confounding effects from temporal invariance were avoided by training and testing the network on temporally normalized utterances, by warping them to the duration of the template utterance for the respective digit. Responses to standard natural test utterances (those not presented during training) were contrasted with two types of artificially created utterances: those with shuffled spectral noise and orthogonal spectral noise (Figure 6B). Shuffled spectral noise was created by shuffling the PCA axes (basis) of the spectral noise (Materials and Methods). In other words, novel utterances were generated by reordering the directions of spectral noise in phase space (Figure 6C). This procedure scrambled the relationships between input signal and noise, while constraining the artificial spectral noise to the same subspace as natural spectral noise. Orthogonal spectral noise was composed of linear bases that were orthogonal to natural spectral noise (Figure 6D, see Materials and Methods). In addition to scrambling the input signal-noise relationships, this procedure also scrambled the relationships between input channels. Care was taken while constructing these artificial inputs, to ensure that the temporal structure, dimensionality and magnitude of the spectral noise were identical to that observed in natural utterances.

Measurements of the within-digit trajectory distances revealed a progressive increase from trained utterances, to untrained utterances, to utterances with shuffled noise, to utterances with orthogonal noise (Figure 6E). In contrast, the untrained (reservoir) network exhibited deviations that were large and fairly similar across all conditions, implying that the trained network was able to identify and suppress natural spectral noise present in the utterances, but not artificial noise of similar magnitude and structure. We further dissected these results with a novel analysis based on the decomposition of the RNN drive into its input and recurrent components, which allowed trajectories to be decomposed into the subspaces wherein the network integrates external (input subspace) and recurrent (recurrent subspace) inputs (Materials and Methods). The analysis showed that in untrained networks, both natural and artificial spectral noise introduced deviations in the recurrent subspace that were large and unsuppressed (Figure 6F). In contrast, in trained RNNs the recurrent dynamics suppressed natural spectral noise, but not artificial noise (Figure 6G). Training also resulted in a recurrent subspace that suppressed spectral noise induced deviations in the input subspace. Whereas the untrained network integrated recurrent and external inputs in orthogonal subspaces, training the network rotated the recurrent subspace such that spectral noise induced trajectory deviations in the recurrent subspace were anti-correlated with the deviations in the input subspace, thereby resulting in their suppression (Recurrent-Input interaction in Figure 6F–G).

Together, these results suggest that recurrent plasticity may play a crucial role in identifying and suppressing the natural spatial variations of stimuli. In our model, this is achieved via supervised training with a common within-digit target trajectory, which exposes the spatiotemporal distribution of the spectral noise in the sensory input, and in doing so, allows for effective sampling from this distribution. Training then shapes the basins of attraction or hypertubes around the spoken digit-encoding trajectories based on this distribution, resulting in effective, recurrently-driven suppression of naturally occurring spectral noise.

Encoding stimuli as neural trajectories allows for temporal generalization (Scaling)

As mentioned above, a signature and little understood feature of how the brain processes time-varying patterns pertains to temporal warping (temporal invariance), whereby temporally compressed or dilated input signals can be identified as the same pattern. Thus, an important test for computational models of sensory processing of time-varying stimuli, is whether they are able to account for temporal invariance. We hypothesized that one advantage of encoding time-varying stimuli as continuous neural trajectories, is that it naturally addresses the problem of temporal warping; specifically, that compressed or dilated stimuli generate similar neural trajectories that play out at different ‘speeds’ (Lerner et al., 2014; Mello et al., 2015). To test this hypothesis, we trained and tested an RNN on a dataset of temporally warped spoken digits.

Specifically, the input pattern, y(t), of a single utterance of each digit from a speaker of the TI-46 dataset was artificially stretched or compressed by a time warp factor α (y($\frac{1}{\alpha }$t)), while retaining its spatial structure. Figure 7A (left panel) shows the input structure for two utterances of the digit ‘nine’, one warped to twice (α = 2, 2x or 200% warp) and the other to half (α = 0.5, 0.5x or 50% warp) the duration of the original utterance.

Figure 7

Download asset Open asset

The RNN was trained on three warped utterances of each digit (warp factors of 0.7, 1 and 1.4) and tested on warp factors in the range 0.5 to 2 (a four-fold factor that approximates the natural speed variation of speech). To discern the effect of a temporally warped input on the encoding trajectory, we constructed a matrix of the distances between trajectories produced by the reference (α = 1) and a warped utterance (Figure 7A). Each element of such a matrix measures the distance between the two trajectories at a corresponding pair of time points. In the case of perfect warping (e.g., when the reference trajectory and the trajectory at an α≠1 overlap exactly in phase space), a diagonal line of zero distances (deep blue) would be observed during the sensory epoch (shaded region), with slope proportional to the warp factor. Trajectories produced by temporally warped inputs in the sensorimotor trained RNN were closer to the reference compared to their counterparts in the reservoir or motor-trained RNN. In other words, tuning the recurrent weights dramatically improved the ability of the network to reproduce the same neural trajectories, despite being driven by inputs at different speeds. We quantified this effect by measuring the average distance between the reference and warped trajectories during the sensory epoch over different warp factors (Figure 7B). In comparison to the two control networks, the trained RNN maintains a much smaller distance between trajectories across the four-fold range of temporal warping—including warp factors outside the range used during training. These results confirm the hypothesis that training produces a modulation of the internal dynamics by the external input that renders the encoding trajectories invariant to temporal warping of the input patterns. The impact of this result is borne out in the network’s performance on the digit transcription task, as measured by the CNN classifier (Figure 7C). Training during the sensory and motor epochs dramatically improves ‘extrapolation’—that is generalization to speeds outside the range of training speeds; however, training during the motor epoch alone is sufficient to produce very good ‘interpolation’ (novel speeds within the training set range) generalization.

Mechanisms underlying temporal generalization (temporal scaling)

It is surprising that a RNN with strong recurrent dynamics can represent temporally scaled stimuli with similar trajectories. Specifically, the strong intrinsic dynamics of the network (governed by the recurrent weights) must somehow match the temporal scale of the input. One possibility is that the network achieves this by scaling the linear speed of its trajectories (i.e. the magnitude of $\frac{d\mathit{x}}{dt}$ in Equation 1) in inverse proportion to the warp factor. However, a comparison of the time-averaged linear speed of trajectories encoding digits at different warps deviates from this relationship in both the untrained and trained networks (Figure 8A), which, in contrast to our earlier results (Figures 4A and 7A), suggests that the trajectories may not be temporally invariant. This contradiction arises from the incorrect assumption that the trajectories have little or no curvature. Figure 8B schematizes the two alternatives for temporal scaling of trajectories with curvature: constant speed/variable distance traversed, versus variable speed/constant distance traversed. At one extreme, a reference trajectory (black curve) can be slowed down while conserving its linear speed by increasing its radius of curvature, and therefore the distance it traverses through phase space (green curve). At the other extreme, the reference may be slowed by appropriately scaling down its linear speed while conserving its radius of curvature (yellow curve), thereby generating temporally invariant trajectories that traverse identical distances.

Figure 8 with 2 supplements see all

Download asset Open asset

Temporally-scaled utterances produce overlapping trajectories in PCA subspace (Figure 8C, left panel)—reflecting the scaling of their linear velocities while traversing constant distances (a direct consequence of how the artificially warped utterances were constructed). In contrast, the RNN trajectories produced by these scaled utterances are spatially distinct with warp-dependent separations (Figure 8C, right panel). Moreover, they exhibit warp-dependent curvature, with slower trajectories incurring larger curvature radii. Together with the results in Figure 8A, this suggests that the mechanisms underlying temporal scaling in our model fall between the two extreme hypotheses, implying that: (i) that the average linear speed and curvature radius of trajectories encoding utterances at different warps adjust in a manner that appropriately modulates their angular speed and therefore their timescale; and (ii) that the trajectories are parallel to each other, demonstrating that they encode the digit similarly. Figure 8D presents measurements of the time-averaged linear speed and the total distance traversed in the recurrent subspace of phase space, by the trajectories in the trained and untrained networks over a range of warps. The latter is a heuristic measure of the time-averaged curvature radius, hinging on the fact that the circumference (or distance traversed) linearly scales with the radius. A comparison of these measurements for the two networks to the expected values in constant speed- and constant distance trajectories, confirms that both networks produce trajectories that lie in between these two extremes. That is, temporal scaling is achieved by an approximate balance of both hypotheses, e.g., at low speeds (digits spoken slowly) the trajectories are slower and longer. Furthermore, training alters this balance to reduce the within-digit separation, by modulating the trajectory speeds more strongly. Measurements to establish whether the trajectories are parallel also agree with these results (Figure 8—figure supplement 1C).

But how do identical inputs (except for their duration) generate these parallel trajectories (e.g., Figure 8C)? To answer this question, we first measured the relationship between the input and recurrent subspaces in the trained network, and discovered that they are orthogonal to one another (data not shown), implying that the integration of the external inputs is independent of the integration of the recurrent inputs. In the absence of subspace interactions, it is easy to see that external inputs with different speeds are integrated into spatially distinct input subspace trajectories (e.g. $\int \cos \left(\omega t\right)dt=\frac{1}{\omega }\sin \left(\omega t\right)+c$, wherein high speed/frequency signals integrate to produce trajectories with small magnitudes) (Figure 8—figure supplement 1A). As with the recurrent subspace trajectories, the resulting input subspace trajectories are parallel to each other and modulate their angular speed via a combination of changes to their curvature radii and linear speed (Figure 8—figure supplement 1A–C). Ultimately, the differentiated curvature and linear velocity that modulate the angular velocity and therefore the timescales of recurrent subspace trajectories, are directly shaped by these input subspace dynamics (Rajan et al., 2010a).

To determine if this solution to the temporal invariance problem is specific to the ‘innate’ training algorithm, or likely a general property of trained RNNs, we trained a RNN with a gradient-descent approach in which the error was based solely on the output units (Materials and Methods, Figure 8—figure supplement 2). While the interpolation and extrapolation performance of this network was inferior to the network trained with ‘innate’ learning (Figure 8—figure supplement 2B), the underlying mechanism was the same: warped utterances were encoded along parallel trajectories (Figure 8—figure supplement 2F–G) with warp-dependent curvature and linear velocity (Figure 8—figure supplement 2E) that subserved temporal scaling. Thus, two very different training algorithms resulted in similar solutions to the temporal invariance problem.

In summary, these results demonstrate that the integration of inputs at different warps separates out the resulting trajectories in the input subspace, and consequently in the recurrent subspace. In both subspaces the network dynamics generate trajectories with warp-dependent linear velocities and curvature, which appropriately modulate their angular velocity, resulting in them traversing spatially distinct yet parallel paths at warp-dependent timescales. This confirms the hypothesis that an inherent computational advantage of encoding time-varying sensory stimuli in neural trajectories is that it can naturally address the temporal invariance problem.

The brain naturally encodes, recognizes, and generates complex time-varying patterns, and can seamlessly process temporally scaled patterns. Despite our poor understanding of these processes, theoretical evidence increasingly suggests that the dynamics of recurrently connected circuits in the brain are critical to representing time-varying patterns. For example, so-called reservoir computing approaches propose that complex high-dimensional spatiotemporal patterns are represented in the dynamics inherent to randomly connected recurrent neural networks (Jaeger and Haas, 2004; Buonomano and Maass, 2009; Sussillo and Abbott, 2009). A shortcoming of this approach, however, is that the recurrent connections in these networks are not plastic; thus in contrast to actual cortical circuits, the random recurrent neural network (the ‘reservoir’) does not adapt or optimize to the task at hand. Additionally, fully tapping into the computational potential of randomly connected RNNs has proven difficult because they are susceptible to chaos (Sompolinsky et al., 1988; Wallace et al., 2013). However, progress has been made on both accounts (Martens and Sutskever, 2011; Vogels et al., 2011; Laje and Buonomano, 2013). Here, we extend these results to better understand well-known aspects of the brain’s ability to represent time-varying sensory and motor patterns—we demonstrate how experience-dependent plasticity could reshape the dynamics of reservoir RNNs to qualitatively improve representations by endowing them with such essential properties as generalization to novel exemplars, sensory-motor association and transformation, and temporal invariance.

Previous models of temporal scaling have enlisted neural and synaptic dynamics to achieve temporal invariance of both sensory stimuli (Gütig and Sompolinsky, 2009) and the generation of motor responses (Murray and Escola, 2017). In the model proposed by Gutig and Sompolinsky, temporally invariant recognition of compressed and dilated sensory stimuli is based on a synaptic shunting mechanism, wherein the effective integration time of a post-synaptic cell is modulated by the total conductance of its afferent synapses, which changes as a monotonic function of the input warp—thus temporal invariance is ultimately reduced to a cellular property. In contrast, in the model described here temporal invariance emerges from a network property: the formation of parallel neural trajectories traversed at different speeds. While the formation of such trajectories may be surprising given the sensitivity and nonlinear nature of RNNs, we show that this capacity is the consequence of scale-dependent angular velocities of the encoding trajectories. This property of RNN encoding, which is observable to a lesser extent even in reservoir networks (Figure 8D), stems from its leaky integration of scaled inputs (Figure 8—figure supplement 1B) and the input-driven suppression of its chaotic recurrent activity (Rajan et al., 2010a). Indeed, network training enhances this property by suppressing recurrent amplification of scale-driven deviations in the encoding trajectories, producing proximal, parallel within-digit trajectories with linear velocities that are more strongly scale-dependent (Figure 8D, Figure 8—figure supplements 1C, 2E and G). The effects of such a suppression are evident even in networks subjected to motor-only training, where we observe a considerable decrease in the within-digit distances (Figure 7B) and consequently strong interpolation generalization (Figure 7C). Nevertheless, it remains the case that training across both the sensory and motor epochs, not only improves interpolation and extrapolation generalization, but also strongly enhances cross-utterance and cross-speaker generalization (Figure 1C).

Our results also demonstrate how training an RNN reveals a computationally powerful pattern recognition regime, one that helps generalize the encoding of learned time-varying patterns to novel exemplars. As stated earlier, recognizing different instances of the same digit as one and the same, while discriminating between different digits requires that networks actively and differentially sculpt the within and between class trajectories—the separation between trajectories representing similar patterns must be kept to a minimum, while that between trajectories representing different patterns must be amplified. A previous study has analytically shown that untrained random recurrent networks (reservoir networks) process inputs in one of three computational regimes: a chaotic regime, wherein any separation between trajectories is strongly amplified by the network’s chaotic dynamics; a regime with weak recurrent weights that is input-dominated with little computational power; and a critical regime wherein the input and internal dynamics are balanced (Bertschinger and Natschläger, 2004). Here, we show that training alters these regimes in very different ways (Figure 5). Crucially, relative to the separation between the input patterns, reservoir networks in the critical regime amplify the separation between trajectories representing similar patterns more strongly than between different patterns. In contrast, trained networks separate trajectories according to the patterns they are representing (Figure 5A). Tuning the recurrent weights reshapes the internal dynamics of the network, enabling it to redirect or suppress deviations from the target trajectories (Figure 6). Ultimately, it is this property of trained networks that allows them to encode complex sensory patterns and effectively generalize across natural utterances and speakers.

Three experimentally tractable predictions emerge from the above results. First, neural trajectories will be stable in response to local perturbations potentially administered through optogenetic stimulation. Importantly, and in agreement with an earlier analysis (Rajan et al., 2010a), the neural trajectories elicited during sensory stimuli will be more resistant to perturbations than the neural trajectories unfolding during the motor epoch of sensory-motor tasks. A second prediction, which to the best of our knowledge is a novel one, relates to the structure of noise. Specifically, trained networks fail at generalizing to unfamiliar spectral noise patterns, thereby generating the prediction that trajectories in sensory areas will diverge more when presented stimuli composed of artificial or unfamiliar spectral noise patterns. This would indicate that the stimulus-response function of complex sensory neurons are not just sensitive to natural stimulus features (Theunissen and Elie, 2014), but also natural noise features. To test this, an animal could be trained to recognize natural sounds with noise injected along some principal components of the stimulus spectrogram but not others, and then tested on noise injected along the held-out PCA dimensions. A comparison of the trajectories encoding trained and tested sounds should reveal a larger divergence in the neural encoding of the test sounds. Third, and most specifically, the model predicts a clear neural correlate of the encoding of temporally scaled stimuli—the observation that slower stimuli yield trajectories with larger curvature radii implies that the neural population activity should exhibit larger fluctuations in their firing rates in response to slower stimuli; In other words, at the neuronal level the range of minimal to maximal firing rate should be larger (Figure 8—figure supplements 1D and 2D).

These experimental predictions are critical to validate the model’s implementation of complex and invariant sensorimotor computations as stable neural trajectories. However, even if validated a number questions remain to be addressed. Most notably, how can the recurrent synaptic strengths be tuned to develop stable trajectories in a biologically plausible manner? The learning rule used here, coupled with its requirement of an explicit target for each recurrent unit, make it biologically implausible. However, while it is important that the sensorimotor representations are encoded as locally stable trajectories, the structure of the target trajectories themselves are essentially arbitrary. It is therefore conceivable that arbitrary yet locally-stable encoding trajectories may emerge from unsupervised learning. A second related issue is that to achieve strong temporal invariance, our model had to be trained over a range of sensory stimuli speeds. Again, it is not clear if this would represent a biologically plausible scenario—to help address this question, it will be important for future research to determine if the ability to recognize stimuli independent of speed is learned through experience. Finally, questions relating to the learning capacity of networks capable of strong temporal invariance, and the expedience of sensory-epoch training for temporal invariance remain open.

The computational potential of continuous time RNNs in high-gain regimes has long been recognized, but it has been challenging to tap into this potential because of their inherently chaotic behavior and the difficulties in training them (Bengio et al., 1994; Pearlmutter, 1995). Step-by-step, progress has been made on how to capture the computational potential of RNNs (Jaeger and Haas, 2004; Sussillo and Abbott, 2009; Martens and Sutskever, 2011; Laje and Buonomano, 2013). Here, we establish that a further feature of trained RNNs is their ability to not only encode spatiotemporal objects, but also to perform complex sensorimotor tasks and address the long-standing problem of temporal warping. These results are consistent with the proposal that that spatiotemporal objects are not encoded as fixed-point attractors, but as locally stable neural trajectories (dynamic attractors).

1. Preprint

   1. Abadi M
   2. Agarwal A
   3. Barham P
   4. Brevdo E
   5. Chen Z
   6. Citro C
   7. Corrado GS
   8. Davis A
   9. Dean J
   10. Devin M

   (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems

   arXiv.

   https://arxiv.org/abs/1603.04467

   • Google Scholar
2. 1. Abbott LF
   2. DePasquale B
   3. Memmesheimer RM

   (2016) Building functional networks of spiking model neurons

   Nature Neuroscience 19:350–355.

   https://doi.org/10.1038/nn.4241

   • PubMed
   • Google Scholar
3. 1. Amit DJ
   2. Brunel N

   (1997) Model of global spontaneous activity and local structured activity during delay periods in the cerebral cortex

   Cerebral Cortex 7:237–252.

   https://doi.org/10.1093/cercor/7.3.237

   • PubMed
   • Google Scholar
4. 1. Ayaz A
   2. Saleem AB
   3. Schölvinck ML
   4. Carandini M

   (2013) Locomotion controls spatial integration in mouse visual cortex

   Current Biology 23:890–894.

   https://doi.org/10.1016/j.cub.2013.04.012

   • PubMed
   • Google Scholar
5. 1. Bengio Y
   2. Simard P
   3. Frasconi P

   (1994) Learning long-term dependencies with gradient descent is difficult

   IEEE Transactions on Neural Networks 5:157–166.

   https://doi.org/10.1109/72.279181

   • PubMed
   • Google Scholar
6. 1. Bertschinger N
   2. Natschläger T

   (2004) Real-time computation at the edge of chaos in recurrent neural networks

   Neural Computation 16:1413–1436.

   https://doi.org/10.1162/089976604323057443

   • PubMed
   • Google Scholar
7. 1. Buonomano DV
   2. Maass W

   (2009) State-dependent computations: spatiotemporal processing in cortical networks

   Nature Reviews Neuroscience 10:113–125.

   https://doi.org/10.1038/nrn2558

   • PubMed
   • Google Scholar
8. 1. Buonomano DV
   2. Merzenich MM

   (1998) Cortical plasticity: from synapses to maps

   Annual Review of Neuroscience 21:149–186.

   https://doi.org/10.1146/annurev.neuro.21.1.149

   • PubMed
   • Google Scholar
9. 1. Carnevale F
   2. de Lafuente
   3. Romo R
   4. Barak O
   5. Parga N

   (2015) Dynamic control of response criterion in premotor cortex during perceptual detection under temporal uncertainty

   Neuron 86:1067–1077.

   https://doi.org/10.1016/j.neuron.2015.04.014

   • PubMed
   • Google Scholar
10. 1. Chang EF
    2. Niziolek CA
    3. Knight RT
    4. Nagarajan SS
    5. Houde JF

    (2013) Human cortical sensorimotor network underlying feedback control of vocal pitch

    PNAS 110:2653–2658.

    https://doi.org/10.1073/pnas.1216827110

    • PubMed
    • Google Scholar
11. 1. Cheung C
    2. Hamiton LS
    3. Johnson K
    4. Chang EF

    (2016) The auditory representation of speech sounds in human motor cortex

    eLife 5:e12577.

    https://doi.org/10.7554/eLife.12577

    • PubMed
    • Google Scholar
12. 1. Crist RE
    2. Li W
    3. Gilbert CD

    (2001) Learning to see: experience and attention in primary visual cortex

    Nature Neuroscience 4:519–525.

    https://doi.org/10.1038/87470

    • PubMed
    • Google Scholar
13. 1. Crowe DA
    2. Zarco W
    3. Bartolo R
    4. Merchant H

    (2014) Dynamic representation of the temporal and sequential structure of rhythmic movements in the primate medial premotor cortex

    Journal of Neuroscience 34:11972–11983.

    https://doi.org/10.1523/JNEUROSCI.2177-14.2014

    • PubMed
    • Google Scholar
14. 1. Doupe AJ
    2. Kuhl PK

    (1999) Birdsong and human speech: common themes and mechanisms

    Annual Review of Neuroscience 22:567–631.

    https://doi.org/10.1146/annurev.neuro.22.1.567

    • PubMed
    • Google Scholar
15. 1. Elman JL

    (1990) Finding structure in time

    Cognitive Science 14:179–211.

    https://doi.org/10.1207/s15516709cog1402_1

    • Google Scholar
16. 1. Feldman DE
    2. Brecht M

    (2005) Map plasticity in somatosensory cortex

    Science 310:810–815.

    https://doi.org/10.1126/science.1115807

    • PubMed
    • Google Scholar
17. Software

    1. Goudar V
    2. Buonomano DV

    (2018) SensoriMotor Transcription, version d5f40d2

    GitHub.

    https://github.com/vgoudar/SensoriMotorTranscription
18. Preprint

    1. Graves A

    (2013) Generating sequences with recurrent neural networks

    arXiv.

    https://arxiv.org/abs/1308.0850

    • Google Scholar
19. 1. Gütig R
    2. Sompolinsky H

    (2009) Time-warp-invariant neuronal processing

    PLoS Biology 7:e1000141.

    https://doi.org/10.1371/journal.pbio.1000141

    • PubMed
    • Google Scholar
20. Book

    1. Haykin S

    (2002)

    Adaptive Filter Theory

    Upper Saddle River: Prentice Hall.

    • Google Scholar
21. 1. Hinton G
    2. Deng L
    3. Yu D
    4. Dahl G
    5. Mohamed A-rahman
    6. Jaitly N
    7. Senior A
    8. Vanhoucke V
    9. Nguyen P
    10. Sainath T
    11. Kingsbury B

    (2012) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups

    IEEE Signal Processing Magazine 29:82–97.

    https://doi.org/10.1109/MSP.2012.2205597

    • Google Scholar
22. 1. Hopfield JJ
    2. Tank DW

    (1986) Computing with neural circuits: a model

    Science 233:625–633.

    https://doi.org/10.1126/science.3755256

    • PubMed
    • Google Scholar
23. 1. Hopfield JJ

    (1982) Neural networks and physical systems with emergent collective computational abilities

    PNAS 79:2554–2558.

    https://doi.org/10.1073/pnas.79.8.2554

    • PubMed
    • Google Scholar
24. 1. Hopfield JJ

    (2015) Understanding emergent dynamics: Using a collective activity coordinate of a neural network to recognize time-varying patterns

    Neural Computation 27:2011–2038.

    https://doi.org/10.1162/NECO_a_00768

    • PubMed
    • Google Scholar
25. 1. Ivry RB
    2. Schlerf JE

    (2008) Dedicated and intrinsic models of time perception

    Trends in Cognitive Sciences 12:273–280.

    https://doi.org/10.1016/j.tics.2008.04.002

    • PubMed
    • Google Scholar
26. 1. Jaeger H
    2. Haas H

    (2004) Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication

    Science 304:78–80.

    https://doi.org/10.1126/science.1091277

    • PubMed
    • Google Scholar
27. Preprint

    1. Jia Y
    2. Shelhamer E
    3. Donahue J
    4. Karayev S
    5. Long J
    6. Girshick R
    7. Guadarrama S
    8. Darrell T

    (2014) Caffe: Convolutional Architecture for Fast Feature Embedding

    arXiv.

    https://arxiv.org/abs/1408.5093

    • Google Scholar
28. 1. Karmarkar UR
    2. Dan Y

    (2006) Experience-dependent plasticity in adult visual cortex

    Neuron 52:577–585.

    https://doi.org/10.1016/j.neuron.2006.11.001

    • PubMed
    • Google Scholar
29. Preprint

    1. Kingma D
    2. Ba J

    (2014) Adam: A method for stochastic optimization

    arXiv.

    https://arxiv.org/abs/1412.6980

    • Google Scholar
30. 1. Laje R
    2. Buonomano DV

    (2013) Robust timing and motor patterns by taming chaos in recurrent neural networks

    Nature Neuroscience 16:925–933.

    https://doi.org/10.1038/nn.3405

    • PubMed
    • Google Scholar
31. Preprint

    1. Le QV
    2. Jaitly N
    3. Hinton GE

    (2015) A simple way to initialize recurrent networks of rectified linear units

    arXiv.

    https://arxiv.org/abs/1504.00941

    • Google Scholar
32. 1. LeCun Y
    2. Bengio Y
    3. Hinton G

    (2015) Deep learning

    Nature 521:436–444.

    https://doi.org/10.1038/nature14539

    • PubMed
    • Google Scholar
33. 1. Lecun Y
    2. Bottou L
    3. Bengio Y
    4. Haffner P

    (1998) Gradient-based learning applied to document recognition

    Proceedings of the IEEE 86:2278–2324.

    https://doi.org/10.1109/5.726791

    • Google Scholar
34. Data

    1. LeCun Y
    2. Cortes C
    3. Burges CJC

    (authors) (1998) Training sets

    The MNIST Database of Handwritten Digits. ID MNIST.

    http://yann.lecun.com/exdb/mnist/
35. 1. Lerner Y
    2. Honey CJ
    3. Katkov M
    4. Hasson U

    (2014) Temporal scaling of neural responses to compressed and dilated natural speech

    Journal of Neurophysiology 111:2433–2444.

    https://doi.org/10.1152/jn.00497.2013

    • PubMed
    • Google Scholar
36. 1. Li N
    2. Daie K
    3. Svoboda K
    4. Druckmann S

    (2016) Robust neuronal dynamics in premotor cortex during motor planning

    Nature 532:459–464.

    https://doi.org/10.1038/nature17643

    • PubMed
    • Google Scholar
37. 1. Lukoševičius M
    2. Jaeger H

    (2009) Reservoir computing approaches to recurrent neural network training

    Computer Science Review 3:127–149.

    https://doi.org/10.1016/j.cosrev.2009.03.005

    • Google Scholar
38. Conference

    1. Lyon RF

    (1982)

    A Computational Model of Filtering, Detection, and Compression in the Cochlea

    Acoustics, Speech, and Signal Processing, IEEE International Conference on ICASSP. pp. 1282–1285.

    • Google Scholar
39. 1. Maass W
    2. Natschläger T
    3. Markram H

    (2002) Real-time computing without stable states: a new framework for neural computation based on perturbations

    Neural Computation 14:2531–2560.

    https://doi.org/10.1162/089976602760407955

    • PubMed
    • Google Scholar
40. 1. Mante V
    2. Sussillo D
    3. Shenoy KV
    4. Newsome WT

    (2013) Context-dependent computation by recurrent dynamics in prefrontal cortex

    Nature 503:78–84.

    https://doi.org/10.1038/nature12742

    • PubMed
    • Google Scholar
41. Data

    1. Mark Liberman RA
    2. Church K
    3. Fox E
    4. Hafner C
    5. Klavans J
    6. Marcus M
    7. Mercer B
    8. Pedersen J
    9. Roossin P
    10. Don Walker SW
    11. Zampolli A

    (authors) (1993) TI 46-Word LDC93S9

    Linguistic Data Consortium. ID LDC93S9.

    https://catalog.ldc.upenn.edu/ldc93s9
42. Conference

    1. Martens J
    2. Sutskever I

    (2011)

    Learning Recurrent Neural Networks with Hessian-Free Optimization

    The International Conference on Machine Learning.

    • Google Scholar
43. 1. Mauk MD
    2. Buonomano DV

    (2004) The neural basis of temporal processing

    Annual Review of Neuroscience 27:307–340.

    https://doi.org/10.1146/annurev.neuro.27.070203.144247

    • PubMed
    • Google Scholar
44. 1. Mello GB
    2. Soares S
    3. Paton JJ

    (2015) A scalable population code for time in the striatum

    Current Biology 25:1113–1122.

    https://doi.org/10.1016/j.cub.2015.02.036

    • PubMed
    • Google Scholar
45. 1. Merchant H
    2. Harrington DL
    3. Meck WH

    (2013) Neural basis of the perception and estimation of time

    Annual Review of Neuroscience 36:313–336.

    https://doi.org/10.1146/annurev-neuro-062012-170349

    • PubMed
    • Google Scholar
46. 1. Miller JL
    2. Grosjean F
    3. Lomanto C

    (1984) Articulation rate and its variability in spontaneous speech: a reanalysis and some implications

    Phonetica 41:215–225.

    https://doi.org/10.1159/000261728

    • PubMed
    • Google Scholar
47. 1. Mnih V
    2. Kavukcuoglu K
    3. Silver D
    4. Rusu AA
    5. Veness J
    6. Bellemare MG
    7. Graves A
    8. Riedmiller M
    9. Fidjeland AK
    10. Ostrovski G
    11. Petersen S
    12. Beattie C
    13. Sadik A
    14. Antonoglou I
    15. King H
    16. Kumaran D
    17. Wierstra D
    18. Legg S
    19. Hassabis D

    (2015) Human-level control through deep reinforcement learning

    Nature 518:529–533.

    https://doi.org/10.1038/nature14236

    • PubMed
    • Google Scholar
48. 1. Murray JM
    2. Escola GS

    (2017) Learning multiple variable-speed sequences in striatum via cortical tutoring

    eLife 6:e26084.

    https://doi.org/10.7554/eLife.26084

    • PubMed
    • Google Scholar
49. 1. Nobre A
    2. Correa A
    3. Coull J

    (2007) The hazards of time

    Current Opinion in Neurobiology 17:465–470.

    https://doi.org/10.1016/j.conb.2007.07.006

    • PubMed
    • Google Scholar
50. 1. Ostojic S

    (2014) Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons

    Nature Neuroscience 17:594–600.

    https://doi.org/10.1038/nn.3658

    • PubMed
    • Google Scholar
51. 1. Pearlmutter BA

    (1995) Gradient calculations for dynamic recurrent neural networks: a survey

    IEEE Transactions on Neural Networks 6:1212–1228.

    https://doi.org/10.1109/72.410363

    • Google Scholar
52. 1. Rabiner LR

    (1989) A tutorial on hidden Markov models and selected applications in speech recognition

    Proceedings of the IEEE 77:257–286.

    https://doi.org/10.1109/5.18626

    • Google Scholar
53. 1. Rabinovich M
    2. Huerta R
    3. Laurent G

    (2008) NEUROSCIENCE: transient dynamics for neural processing

    Science 321:48–50.

    https://doi.org/10.1126/science.1155564

    • Google Scholar
54. Conference

    1. Rajan K
    2. Abbott L
    3. Sompolinsky H

    (2010b) Inferring Stimulus Selectivity from the Spatial Structure of Neural Network Dynamics

    Advances in Neural Information Processing Systems 23 (NIPS 2010).

    https://papers.nips.cc/paper/4051-inferring-stimulus-selectivity-from-the-spatial-structure-of-neural-network-dynamics

    • Google Scholar
55. 1. Rajan K
    2. Abbott LF
    3. Sompolinsky H

    (2010a) Stimulus-dependent suppression of chaos in recurrent neural networks

    Physical Review E 82:011903.

    https://doi.org/10.1103/PhysRevE.82.011903

    • PubMed
    • Google Scholar
56. 1. Rajan K
    2. Abbott LF

    (2006) Eigenvalue spectra of random matrices for neural networks

    Physical Review Letters 97:188104.

    https://doi.org/10.1103/PhysRevLett.97.188104

    • PubMed
    • Google Scholar
57. 1. Rajan K
    2. Harvey CD
    3. Tank DW

    (2016) Recurrent network models of sequence generation and memory

    Neuron 90:128–142.

    https://doi.org/10.1016/j.neuron.2016.02.009

    • PubMed
    • Google Scholar
58. 1. Schneider DM
    2. Mooney R

    (2015) Motor-related signals in the auditory system for listening and learning

    Current Opinion in Neurobiology 33:78–84.

    https://doi.org/10.1016/j.conb.2015.03.004

    • PubMed
    • Google Scholar
59. 1. Sebastián-Gallés N
    2. Dupoux E
    3. Costa A
    4. Mehler J

    (2000) Adaptation to time-compressed speech: phonological determinants

    Perception & Psychophysics 62:834–842.

    https://doi.org/10.3758/BF03206926

    • PubMed
    • Google Scholar
60. 1. Sompolinsky H
    2. Crisanti A
    3. Sommers HJ

    (1988) Chaos in random neural networks

    Physical Review Letters 61:259–262.

    https://doi.org/10.1103/PhysRevLett.61.259

    • PubMed
    • Google Scholar
61. 1. Sussillo D
    2. Abbott LF

    (2009) Generating coherent patterns of activity from chaotic neural networks

    Neuron 63:544–557.

    https://doi.org/10.1016/j.neuron.2009.07.018

    • PubMed
    • Google Scholar
62. 1. Theunissen FE
    2. Elie JE

    (2014) Neural processing of natural sounds

    Nature Reviews Neuroscience 15:355–366.

    https://doi.org/10.1038/nrn3731

    • PubMed
    • Google Scholar
63. 1. Vogels TP
    2. Sprekeler H
    3. Zenke F
    4. Clopath C
    5. Gerstner W

    (2011) Inhibitory plasticity balances excitation and inhibition in sensory pathways and memory networks

    Science 334:1569–1573.

    https://doi.org/10.1126/science.1211095

    • PubMed
    • Google Scholar
64. 1. Waibel A
    2. Hanazawa T
    3. Hinton G
    4. Shikano K
    5. Lang KJ

    (1989) Phoneme recognition using time-delay neural networks

    IEEE Transactions on Acoustics, Speech, and Signal Processing 37:328–339.

    https://doi.org/10.1109/29.21701

    • Google Scholar
65. 1. Wallace E
    2. Maei HR
    3. Latham PE

    (2013) Randomly connected networks have short temporal memory

    Neural Computation 25:1408–1439.

    https://doi.org/10.1162/NECO_a_00449

    • PubMed
    • Google Scholar
66. 1. Wang XJ

    (2001) Synaptic reverberation underlying mnemonic persistent activity

    Trends in Neurosciences 24:455–463.

    https://doi.org/10.1016/S0166-2236(00)01868-3

    • PubMed
    • Google Scholar

Author details

1. Vishwa Goudar

   Departments of Neurobiology, University of California, Los Angeles, Los Angeles, United States

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Writing—original draft, Writing—review and editing

   For correspondence

   vishwa.goudar@gmail.com

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-6612-3076

2. Dean V Buonomano

   1. Departments of Neurobiology, University of California, Los Angeles, Los Angeles, United States
   2. Integrative Center for Learning and Memory, University of California, Los Angeles, Los Angeles, United States
   3. Departments of Psychology, University of California, Los Angeles, Los Angeles, United States

   Contribution

   Conceptualization, Supervision, Funding acquisition, Writing—original draft, Writing—review and editing

   For correspondence

   dbuono@ucla.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-8528-9231

• 4,467

  views

• 606

  downloads

• 67

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.