Our ability to perceive the world is dependent on information from our senses being passed between different parts of the brain. The information is encoded as patterns of electrical pulses or ‘spikes’, which other brain regions must be able to decipher. Cracking this code would thus enable us to predict the patterns of nerve impulses that would occur in response to specific stimuli, and ‘decode’ which stimuli had produced particular patterns of impulses.

This task is challenging in part because of its scale—vast numbers of stimuli are encoded by huge numbers of neurons that can send their spikes in many different combinations. Furthermore, neurons are inherently noisy and their response to identical stimuli may vary considerably in the number of spikes and their timing. This means that the brain cannot simply link a single unchanging pattern of firing with each stimulus, because these firing patterns are often distorted by biophysical noise.

Ganmor et al. have now modeled the effects of noise in a network of neurons in the retina (found at the back of the eye), and, in doing so, have provided insights into how the brain solves this problem. This has brought us a step closer to cracking the neural code. First, 10 second video clips of natural scenes and artificial stimuli were played on a loop to a sample of retina taken from a salamander, and the responses of nearly 100 neurons in the sample were recorded for two hours. Dividing the 10 second clip into short segments provided a series of 500 stimuli, which the network had been exposed to more than 600 times.

Ganmor et al. analyzed the responses of groups of 20 cells to each stimulus and found that physically similar firing patterns were not particularly likely to encode the same stimulus. This can be likened to the way that words such as ‘light’ and ‘night’ have similar structures but different meanings. Instead, the model reveals that each stimulus was represented by a cluster of firing patterns that bore little physical resemblance to one another, but which nevertheless conveyed the same meaning. To continue on with the previous example, this is similar to way that ‘light’ and ‘illumination’ have the same meaning but different structures.

Ganmor et al. use these new data to map the organization of the ‘vocabulary’ of populations of cells the retina, and put together a kind of ‘thesaurus’ that enables new activity patterns of the retina to be decoded and could be used to crack the neural code. Furthermore, the organization of ‘synonyms’ is strikingly similar to codes that are favored in many forms of telecommunication. In these man-made codes, codewords that represent different items are chosen to be so distinct from each other that even if they were corrupted by noise, they could be correctly deciphered. Correspondingly, in the retina, patterns that carry the same meaning occupy a distinct area, and new patterns can be interpreted based on their proximity to these clusters.

Noise is prevalent in the nervous system, from ion channels through synapses and single neurons, and up to the system level (Mainen and Sejnowski, 1995; Schneidman et al., 1998; Rieke et al., 1999; Osborne et al., 2005; Faisal et al., 2008; Ala-Laurila et al., 2011). Consequently, even when presented with identical stimuli, the brain may receive different spiking patterns from the sensory neurons (Mainen and Sejnowski, 1995; Berry et al., 1997; de Ruyter van Steveninck et al., 1997; Reich et al., 2001). The nature of neural noise thus determines how information is encoded in the brain (Borst and Theunissen, 1999; Stocker and Simoncelli, 2006; Cafaro and Rieke, 2010; Rolls and Treves, 2011), and how it might be read out from neural activity (Warland et al., 1997; Dan et al., 1998; Vargas-Irwin et al., 2010). To correctly decode, the brain must overcome this noise (Osborne et al., 2005; Sreenivasan and Fiete, 2011).

In engineering, codes are designed to solve this problem by choosing codewords that are far apart in the space of patterns, relative to the typical noise levels. That way, if noise corrupts a message, it would still be easily distinguishable from the noisy variants of other codewords (Cover and Thomas, 1991; Sreenivasan and Fiete, 2011; Curto et al., 2013). It is not clear however, how this issue is resolved in the brain, or how it affects the design of the neural code, where information is carried by the joint activity patterns of large groups of noisy neurons (Nicolelis et al., 1995; Maynard et al., 1999; Mazor and Laurent, 2005; Fujisawa et al., 2008; Pillow et al., 2008; Truccolo et al., 2010; Ganmor et al., 2011a; Harvey et al., 2012). It is clear that the nature of correlations among neurons is central in shaping the code's capacity and content, in particular in the context of noise (Zohary et al., 1994; Zemel et al., 1998; Abbott and Dayan, 1999; Diesmann et al., 1999; Sompolinsky et al., 2001; Schneidman et al., 2006; Pillow et al., 2008; Shlens et al., 2009; Ganmor et al., 2011b; Schwartz et al., 2012; Granot-Atedgi et al., 2013). However, the functional role of these correlations in encoding information by large populations has been heavily debated (Nirenberg et al., 2001; Amari et al., 2003; Pola et al., 2003; Schneidman et al., 2003a, 2006; Averbeck et al., 2006; Tkacik et al., 2006; de la Rocha et al., 2007; Pillow et al., 2008; Ecker et al., 2010; Ohiorhenuan et al., 2010; Oizumi et al., 2010; Ganmor et al., 2011a), partly because of the difficulty to study them directly at the network level and the limitations of generalizing from small groups of cells to large ones.

To uncover the structure and content of neural population codebooks, we must be able to quantify the similarity between population activity patterns. In other words, we need to understand network noise well enough to know which patterns are likely to be interchanged in encoding the same stimulus. Several studies suggested intuitive and computationally efficient measures of spike train similarity (Victor and Purpura, 1997; van Rossum, 2001; Houghton and Sen, 2008). However, they mostly focused on single neurons and often assumed the functional form of similarity between responses. Despite their computational simplicity, it is not clear whether such simple metrics can adequately represent similarity between neural responses. The Hamming distance, for example, is a very intuitive measure of similarity between population activity patterns, simply defined as the number of neurons that switch state from spiking to silence and vice versa. Yet, it measures similarity in form, or syntactic similarity, but not necessarily similarity in meaning, that is, semantic similarity (Huth et al., 2012). This would be analogous to suggesting that ‘night’ and ‘light’ are similar in meaning because they differ by just one letter. Extending such metrics to large populations requires additional assumptions on the way the similarity measures between different cells should be combined.

We suggest here that the semantic similarity between neural activity patterns can be measured by comparing what each pattern tells the brain about the stimulus. We demonstrate that in the vertebrate retina, responding to natural and artificial stimuli, we can estimate this similarity between any pair of population responses, using models of population encoding noise. Using this similarity structure, or ‘thesaurus’, we can map the semantic organization and content of the population codebook and show how it enables the accurate decoding of novel population patterns.

To study the variability, or noise, which is inherent to encoding of sensory stimuli by large populations of neurons, we presented isolated salamander retinas with over 600 repeats of the same 10 s long video clips (one natural video and one artificial spatially uniform and temporally Gaussian flicker video), and recorded the joint response of dozens of retinal ganglion cells (see ‘Materials and methods’). Each video was presented continuously in a loop for ∼2 hr. This allowed us to collect hundreds of neural population responses evoked by the exact same visual stimulus (including identical stimulus history up to several minutes). Figure 1A shows examples of the stochastic, or noisy, responses of single neurons from the population to the same stimulus segment (see also Mainen and Sejnowski, 1995; Berry et al., 1997).

Figure 1

Download asset Open asset

In order to examine the responses at the population level, we defined the population activity pattern r(t) as the binary vector of the instantaneous states of each neuron: the ith entry in the vector, rⁱ, being 1 if the ith neuron spiked, and 0 if it was silent (responses were binned in 20 ms bins; for other bin sizes see Figure 5—figure supplement 4). Note that each time point, t, corresponds to a particular segment of the video (t runs from 0 to 10 s—the length of the video), and thus to a particular visual stimulus which we denote as s(t). For a 10 s long video, binned at 20 ms, we have 500 such stimuli.

By collecting the responses recorded at the same time point in the video across many repeated presentations, we obtain samples from the conditional distribution of activity patterns r given a stimulus s, denoted P(r|s). This distribution is commonly referred to as the ‘encoding distribution’ of the stimulus. Figure 1B shows samples of the joint responses of 20 neurons (the population size we focus on in this study) across the repetitions of the video, reflecting the wide variety of different population activity patterns evoked by the same stimulus.

We quantified the variability of the population patterns that encoded the same stimulus by the entropy, H, of the encoding distribution, P(r|s), which measures the ‘width’ or richness of the distribution (see ‘Materials and methods’). Figure 1C shows the entropy of P(r|s) as a function of time (or, equivalently, of stimulus), along with the instantaneous population rate, reflecting that the variability of the population responses is high when the population's firing rate goes up.

Population responses to ‘interesting’ stimuli exhibit strong network noise correlations

The properties of encoding noise, and in particular, the magnitude and importance of correlations between neurons in encoding a stimulus, commonly known as ‘noise correlations’, have been heavily debated (Nirenberg et al., 2001; Schneidman et al., 2003a; Ohiorhenuan et al., 2010; Oizumi et al., 2010). For pairs of neurons, average noise correlations are typically weak (Cafaro and Rieke, 2010; Ecker et al., 2010), implying that pairs of neurons are not far from being conditionally independent given the stimulus. If groups of cells were encoding information independently, that is, if $P\left(r|s\right)={\Pi }_{i}P\left(r^{\text{i}}|s\right)$ for large populations, then P(r|s) would be defined by the individual and independent noise of each neuron and would be easily learned from the noise of each neuron P(rⁱ|s). However, weak pairwise correlations do not imply that large groups are conditionally independent in encoding stimuli (Schneidman et al., 2006; Pillow et al., 2008; Vidne et al., 2012; Granot-Atedgi et al., 2013). Moreover, noise correlations are often estimated on average, over a range of different stimuli, and it is not clear what low average noise correlations imply for population encoding of specific stimuli.

We therefore estimated the noise correlations at the population level, for each stimulus s. Specifically, we quantified the correlation of the population in encoding the stimulus s, by the multi-information, $I\left(r|s\right)=H\left[P^{ind}\hspace{0.17em}\left(r|s\right)\right]-H\left[P\left(r|s\right)\right]$ (Amari, 2001; Schneidman et al., 2003b), where $H\left[P^{ind}\hspace{0.17em}\left(r|s\right)\right]$ is the entropy assuming neurons are independent given the stimulus—$P^{\text{ind}}\left(r|s\right)={\Pi }_{i}P\left(r^{\text{i}}|s\right)$, and $H\left[P\left(r|s\right)\right]$ is the entropy of the joint population response to the stimulus s (estimated following Strong et al., 1998). We note that I(r|s) measures the total correlations of all orders among the cells. We found that most stimuli did not evoke a substantial response from the retina, which gave rise to low noise correlation in the network for these stimuli. However, when something ‘interesting’ happened in the video and the ganglion cells increased their firing rates, we found a sharp increase in the degree of network noise correlations for both natural and artificial videos (Figure 2A,B; see Figure 2—figure supplement 1 for analysis of sampling properties). Thus, while on average the network noise correlations may be weak, the population was strongly correlated and far from conditionally independent in response to interesting stimuli. In other words, for these stimuli, the variability or noise at the level of the network is significantly reduced compared to what would be expected from the apparent noise level of individual cells.

Figure 2 with 2 supplements see all

Download asset Open asset

Indeed, conditionally independent models of population encoding (which assume no noise correlations) fail exactly at such interesting stimuli (Pillow et al., 2008; Granot-Atedgi et al., 2013), reflecting that in order to accurately characterize the population responses and the noise, we need a model that takes into account correlations between the cells that depend on the stimulus. We extended the framework of maximum entropy models for neural activity patterns (Schneidman et al., 2006; Shlens et al., 2009), to the population responses to each stimulus (Granot-Atedgi et al., 2013): for each stimulus, s, we fit the minimal model that has the same stimulus dependent firing rates and pairwise noise correlations observed over the repeats of that stimulus. This stimulus-dependent pairwise maximum entropy model is given by

(2) $P^{\left(2\right)}\left(r\text{|}s\right)=\frac{1}{Z\left(s\right)}\text{exp}\left({\displaystyle \sum }_i{\alpha }_i\left(s\right)r^i+{\displaystyle \sum }_{i<j}{\beta }_{ij}\left(s\right)r^i{r}^j\right),$

where α_i(s) and β_ij(s) were fit to obey the constraints. We found that P⁽²⁾(r|s) gave relatively accurate models of the distribution of patterns that the population displayed across repeats, for the different stimuli (Figure 2—figure supplement 2), in agreement with Granot-Atedgi et al. (2013). In particular, they captured accurately the strong network noise correlations (Figure 2C).

A thesaurus for a neural population code

We then use the distributions P⁽²⁾(r|s) that describe the population encoding noise, to estimate the dissimilarity, d(r_i,r_j), between any pair of patterns over the population in a reliable and precise manner. Since there are over a million possible population activity patterns for 20 neurons, we present here only the similarity matrix of all the population patterns that were observed in the test data (half of the data, which was randomly selected and not used to learn the similarity between activity patterns).

If stimuli were represented by overall population rates (Bohte et al., 2000; Amari et al., 2003; Loebel et al., 2007; Schwartz et al., 2012), then responses containing the same number of spiking neurons would be similar, and thus would have a small d between them. Figure 3A shows the similarity matrix d(r_i,r_j) for one representative group of 20 cells, where matrix rows (and columns) stand for individual population patterns and are ordered according to the total number of spikes in each response. The lack of structure in the matrices for both artificial and natural stimuli shows that similar spike counts do not imply similar meaning.

Figure 3 with 2 supplements see all

Download asset Open asset

In clear contrast, when we used a hierarchical clustering algorithm on d(r_i,r_j) matrices and grouped responses together according to their similarity (see ‘Materials and methods’), we found that the ordered matrices exhibit an almost perfect block diagonal structure (Figure 3B; see Figure 3—figure supplement 1 for statistical analysis). Thus, the population code that is used by the retina both for artificial videos, and for natural videos, is arranged in sets, or clusters, of highly similar activity patterns, which are very different from all other patterns. We term these groups of highly similar responses neural ‘synonyms’, and by analogy we refer to the similarity measure d as a neural thesaurus.

We examined whether the Hamming distance, which is a simple and intuitive measure of similarity between population responses (the number of neurons that differ in their spiking or silence state), was sufficient to reveal the structure of the population codebook. We thus constructed the matrices of Hamming distances between all pairs of population activity patterns, and clustered the responses, using the same hierarchical agglomerative clustering. Figure 3C shows the Hamming matrix for the same group of 20 cells from a–d, where matrix rows (and columns) are ordered according to the clustering results. We did not find evident structure in the codebook used for natural stimuli, and only slightly more structure was apparent for the artificial stimuli. We emphasize that the results of Figure 3 were typical for many independent choices of groups of cells (as is summarized later in Figure 5).

These results reflect the importance of measuring similarity in meaning and not similarity in structure. Using a syntactic (structure based) measure, we would not have been able to uncover the clustered organization of the neural population code that the semantic similarity reveals. Patterns that belong to the same cluster do exhibit some shared structure, namely some of the neurons almost always fire in a specific cluster and others are always silent (Figure 3B, bottom). However, importantly, we found that the semantic similarity structure over all the patterns we observed could not be captured by a simple linear or bilinear function of the population patterns (Figure 3—figure supplement 2).

Coding by clusters

The clustered structure of the neural population codebook suggests that the same stimulus may be represented by different, yet semantically similar population patterns, or synonyms. Such structure is commonly used in engineered codes in computer science and communications (Cover and Thomas, 1991; Sreenivasan and Fiete, 2011; Curto et al., 2013). This gives rise to two important predictions, which we confirmed by cross-validation, using novel (held out) test stimuli: 1. Responses to repeated presentations of a stimulus should come from the same cluster. 2. Responses from the same cluster should be nearly interchangeable.

To directly show the advantages of cluster-based encoding of information by the retina, we quantified the reliability of population patterns used to encode the same stimulus. Because of the noise, the responses to repeated presentations of the same stimulus are so variable that even the most frequent population pattern would occur only a handful of times (Figure 4A,B). However, the reliability of the population code is revealed when instead of focusing on individual patterns, we count how many of the population patterns evoked by the same stimulus belong to the same cluster. Notably, even when the population response was highly unreliable (i.e., the most frequent response pattern appeared less than 20% of the time), often over 70% of the observed responses would fall within a single cluster (mostly for the natural video), for the cross-validated test data (see Figure 4—figure supplement 1 for statistical analysis).

Figure 4 with 1 supplement see all

Download asset Open asset

Clusters convey most of the information available about the stimulus

The most direct test of ‘coding by clusters’ is to ask how much information about the stimulus would be lost if instead of knowing the precise population activity pattern (exactly which neurons spiked and which ones were silent) we only knew which cluster the pattern belongs to. We therefore compared the information that the full set of population responses r carry about the stimulus I(s;r), to the information that is carried just by knowing which cluster (out of k possible ones) the response belongs to, I(s;C_k(r)). To that end, we labeled every population pattern in the test data according to its cluster identity, C_k(r), based on the similarity structure learned from training data (see ‘Materials and methods’). To avoid any arbitrary assumptions about the number of clusters, we assessed how much information is carried by k clusters, for different values of k (Figure 5) on novel test data that was not used for learning the similarity structure. We found that ∼100 clusters were enough to account for over 80% of the information available about the stimulus in the detailed population patterns, for both types of stimuli. Importantly, clustering the responses based on the Hamming distance between them gave significantly worse results. This clearly reflects that responses in the same cluster have a similar meaning and can indeed be viewed as noisy variants of a noise-free codeword, similar to the ground states or attractors in a Hopfield model (Hopfield, 1982; Tkacik et al., 2006).

Figure 5 with 4 supplements see all

Download asset Open asset

We therefore conclude that grouping responses by their similarity in the training data identified clusters that conserved the information available about the stimuli in the test data, thus indicating that the similarity we measured is a general feature of the code and not limited to a particular set of stimuli. Importantly, all stimuli in the test data and many of the neural responses they evoked were not observed in the training data that we used to learn d(r_i,r_j). Thus, the limited number of stimuli and responses observed in the training data were sufficient to identify semantic clusters and predict the similarity between over a million possible neural responses evoked by a multitude of different possible stimuli, which were verified using the test data. We further point out that d(r_i,r_j) was so stable across time and for different selections of train and test data, that the information curves derived from clustering the cross-validated test data and the training data itself were nearly identical (Figure 5—figure supplement 1). In addition, the number of clusters required to convey 80% of the information seems to saturate with the size of the test set (Figure 5—figure supplement 1). We conclude that the similarity between neural responses does not rely on the specific stimuli we used and would generalize to other stimuli at least within the same stimulus class.

We further emphasize that clustering of the responses was based on similarity alone and was not optimized to maximize information in any way (which could therefore give even better results). Yet, even if we clustered patterns by explicitly trying to maximize the information, we achieved very similar results (Figure 5—figure supplement 2). This suggests that grouping responses simply by their semantic similarity may be nearly optimal from an information transmission standpoint.

We also estimated the semantic similarity between population patterns with simpler models for the population responses to the stimuli—using conditionally independent models of encoding, where for each stimulus s the response is described by $P\left(r|s\right)={\Pi }_{i}P\left(r^{\text{i}}|s\right)$. Although these models give a different and less accurate estimate for the probability distribution than the pairwise model P⁽²⁾(r|s) (Figure 2), we find that grouping the population activity patterns using this model results in a similar clustered structure (see Figure 5—figure supplement 3). This may suggest that the codebook organization into clusters is sufficiently robust, so that even when using a less accurate model of the neural responses one can identify the organization of the response space (see more in the ‘Discussion’).

Mapping the structure of the neural population codebook and its meaning

Given that almost all information about the stimulus is carried by the identity of the cluster that a given activity pattern belongs to, we asked whether we can map the functional organization of the population codebook. We used Isomap (Tenenbaum et al., 2000) to present a low dimensional embedding of all the population responses associated with clusters that contain 30–300 responses in three-dimensional Euclidean space (Figure 6A; see ‘Materials and methods’ and Video 1 for the raw data points in 3D). Isomap is an embedding algorithm for high dimensional data that preserves the geodesic distance between points. While this embedding is imperfect (one would need more dimensions to achieve a nearly perfect embedding; see Figure 6—figure supplement 1), it provides an approximate view of the organization of the population code of a sensory system and coarsely reflects the ‘cloud’ of patterns that make up each cluster.

Figure 6 with 1 supplement see all

Download asset Open asset

Video 1

Download asset

To show the functional correlates of these clusters in coding, we present the similarity between clusters in terms of the stimuli that they encode. To that end, we quantified the distance between each pair of clusters, C_m and C_l, by the average distance between all inter-cluster response pairs, ${\langle d\left(r_i,r_j\right)\rangle }_{i\in C_1,j\in C_2}$. We then embed the clusters in a Euclidean space according to the similarity between them (again using Isomap). Thus, in Figure 6B each cluster is represented by the average of its associated stimulus ensemble (we emphasize that the stimuli were taken exclusively from the test data). Hence, waveforms that are closer in space belong to clusters that are more similar. We find that response similarity measured from the training data predicts stimulus similarity in the test data (Figure 6—figure supplement 1 shows the correlation between cluster similarity and stimulus similarity; for more accurate metrics on stimulus space, see [Tkačik et al., 2013]). Thus, the similarity measure proposed here for the population responses captures similarity in meaning and generalizes across stimuli. We note that this analysis could only be carried out on the simple one dimensional full field video, as there is no evident way to reduce the dimensionality of natural stimuli.

Decoding novel population patterns using the neural thesaurus

The organization of the retina codebook may also explain how the brain can decode novel stimuli from novel neural activity patterns. Namely, if we observe a response r that we have never seen before, we can now ask what similar responses tell us about the stimulus. Importantly, this can be done since P⁽²⁾(r|s) allows us to estimate d(r_i,r_j) even for patterns we have not seen in the past. Indeed, we found that P(s|r) for a held-out test response, r, could be well estimated simply by taking P(s|r′) for the response, r′, which is most similar to it. Similarity was assessed using the thesaurus that we learned from the training data, that is, for different stimuli than the ones we tested on (Figure 7A,B). This approach clearly improved our ability to estimate the stimulus compared to our prior knowledge about the stimulus, as measured by the Jensen-Shannon divergence between the ‘true’ P(s|r) and the estimate (Figure 7C). Using a thesaurus based on the Hamming distance clearly degraded performance and reduced the accuracy of the stimulus estimate (Figure 7C).

Figure 7 with 1 supplement see all

Download asset Open asset

Thus, using a very limited set of known responses it is possible to decode novel responses to novel stimuli. This may be crucial when encountering new responses to known stimuli, due to sensory noise, or when generalizing prior knowledge to novel stimuli.

We presented a thesaurus for a neural population code—a similarity measure between population activity patterns, based on the meaning of the responses and not on their syntactic structure. This is a general framework for mapping the codebook in any sensory system, since it makes no assumptions about what makes two spike trains similar, or that the similarity structure should obey a metric in the strict mathematical sense. Indeed, it revealed that the neural code of the brain is very different than our usual mathematical intuition: what we may regard as intuitive measures, like the Hamming distance, fail to capture the similarity between neural population responses. Instead, we found a highly structured codebook, organized in groups of responses that carry very similar information, and are distinct from all other responses.

This organization of the neural population codebook into a relatively small number of semantically different clusters is reminiscent of the design of engineered codes used in communication. There, to overcome noise, different codewords are placed far apart in space of codewords, to allow for error-correction. Since noise may corrupt a message that is sent through a channel, when a pattern is received, it is compared to all possible codewords and is classified according to the nearest one. Importantly, the distance between codewords must be defined by the nature of the noise. Here, we found a neural population code that seems to be built along similar lines. Our thesaurus revealed that most of the information that the population encodes is held by the identity of the cluster that a specific activity pattern belongs to, and that more detailed information may be carried by the fine structure of the pattern. These results generalized across novel stimuli and scaled slowly with the number of stimuli used (Figure 5—figure supplement 1). We thus suggest that our analysis reveals a design principle of the population codebook of a sensory circuit.

What are the advantages of such a population code design? We note that for a large neural population, most network activity patterns are observed at most once (Ganmor et al., 2011a). Thus, the brain frequently receives from its sensory neurons activity patterns that were never seen before. This may occur either due to sensory noise, corrupting the response to a known stimulus but also as a result of a truly novel stimulus. In either case, a semantic similarity measure between neural population responses may explain how these novel inputs may be deciphered: when a novel response pattern is encountered it can take on the information content of similar responses, whose meaning is already known. Therefore, the thesaurus may not only explain how sensory noise can be overcome but also how the brain can generalize across stimuli. Indeed, the thesaurus enabled us to decode neural activity patterns we have not seen before, based on their similarity to other responses.

Uncovering the structure and organization of the population code relied on our ability to accurately model the noisy nature of the population response to specific stimuli. It is the nature of the noise that determines how likely two responses are to be interchanged, and consequently the semantic similarity between them. Here, we used pairwise maximum entropy models to capture the ‘noise correlations’ at every time point, (Figure 2), which were weak on average, but significant at specific times, as was also shown in (Kohn and Smith, 2005; Pillow et al., 2008; Cafaro and Rieke, 2010; Granot-Atedgi et al., 2013).

Interestingly, learning a thesaurus using the conditionally independent model of the neural responses to the stimuli, recovered a similarity structure that was on par with that revealed by the pairwise model, in terms of the information that the clusters carried about the stimulus. These results are surprisingly given that the cells were clearly not conditionally independent. One possible explanation is that the organization of the codebook may be so robust, namely that the separation between clusters is sufficiently large, that even an inaccurate model can capture the identity clusters because the errors of the model are smaller than the distance between clusters. The results of (Schwartz et al., 2012; Granot-Atedgi et al., 2013) suggest that the contribution of noise correlations to shaping the population response to stimuli will increase significantly with network size. Since by definition, the conditional independent model implies larger encoding noise (noise entropy), this suggests that the clusters would be ‘wider’ in this case. It would therefore be most interesting to explore the thesaurus that these models would give for larger populations of cells, beyond groups of 20 cells that we used here for sampling reasons. Finally, we note again that the pairwise models gave better results for decoding stimuli over the conditionally independent model.

Previous studies have suggested mathematically elegant and computationally efficient measures of spike train similarity, relying mostly on edit-distance based metrics (Victor and Purpura, 1997; van Rossum, 2001; Victor, 2005; Houghton and Sen, 2008). Our approach is fundamentally different, as it does not assume a metric structure, or particular features of the code (Haslinger et al., 2013) and does not require assumptions about the syntactic similarity between activity patterns. But maybe most importantly, we found that the semantic similarity between population responses could not be approximated by simple linear or bilinear functions of the patterns. This is mostly due to the fact that noise showed a high degree of stimulus dependence (Figure 3—figure supplement 2). This means that the correlated nature of neural noise shapes the population code differently for different stimuli.

The approach we presented here could be immediately applied to other neural circuits (Bathellier et al., 2012; Parnas et al., 2013), and it is therefore important to note where our approach has been restrictive and could be extended. First, our clustering was not optimized to maximize the information that the clusters carry about the stimulus, but only to group similar responses together. Interestingly, such an information maximization clustering approach (Slonim and Tishby, 1999) does not result in significantly more information using fewer clusters (Figure 5—figure supplement 2). Second, hard clustering of the population responses into k clusters is somewhat arbitrary, and it is possible to consider generalizations to fuzzy rather than hard clustering—where responses may be associated with several clusters, or categories, simultaneously in a probabilistic manner (Zemel et al., 1998). Lastly, while both agglomerative and spectral clustering yielded similar results in our analysis of the nature of the retinal population code, it is possible that other ways of clustering would reveal further structure of the code.

1. 1. Abbott LF
   2. Dayan P

   (1999)

   The effect of correlated variability on the accuracy of a population code

   Neural Computation 11:91–101.

   • Google Scholar
2. 1. Ala-Laurila P
   2. Greschner M
   3. Chichilnisky EJ
   4. Rieke F

   (2011) Cone photoreceptor contributions to noise and correlations in the retinal output

   Nature Neuroscience 14:1309–1316.

   https://doi.org/10.1038/nn.2927

   • Google Scholar
3. 1. Amari S

   (2001) Information geometry on hierarchy of probability distributions

   IEEE Transactions on Information Theory 47:1701–1711.

   https://doi.org/10.1109/18.930911

   • Google Scholar
4. 1. Amari S
   2. Nakahara H
   3. Wu S
   4. Sakai Y

   (2003) Synchronous firing and higher-order interactions in neuron pool

   Neural Computation 15:127–142.

   https://doi.org/10.1162/089976603321043720

   • Google Scholar
5. 1. Averbeck BB
   2. Latham PE
   3. Pouget A

   (2006) Neural correlations, population coding and computation

   Nature Reviews Neuroscience 7:358–366.

   https://doi.org/10.1038/nrn1888

   • Google Scholar
6. 1. Bathellier B
   2. Ushakova L
   3. Rumpel S

   (2012) Discrete neocortical dynamics predict behavioral categorization of sounds

   Neuron 76:435–449.

   https://doi.org/10.1016/j.neuron.2012.07.008

   • Google Scholar
7. 1. Berry MJ
   2. Warland DK
   3. Meister M

   (1997) The structure and precision of retinal spike trains

   Proceedings of the National Academy of Sciences of USA 94:5411–5416.

   https://doi.org/10.1073/pnas.94.10.5411

   • Google Scholar
8. 1. Bohte SM
   2. Spekreijse H
   3. Roelfsema PR

   (2000) The effects of pair-wise and higher order correlations on the firing rate of a post-synaptic neuron

   Neural Computation 12:153–179.

   https://doi.org/10.1162/089976600300015934

   • Google Scholar
9. 1. Borst A
   2. Theunissen FE

   (1999) Information theory and neural coding

   Nature Neuroscience 2:947–957.

   https://doi.org/10.1038/14731

   • Google Scholar
10. 1. Cafaro J
    2. Rieke F

    (2010) Noise correlations improve response fidelity and stimulus encoding

    Nature 468:964–967.

    https://doi.org/10.1038/nature09570

    • Google Scholar
11. Book

    1. Cover TM
    2. Thomas JA

    (1991)

    Elements of information theory

    New York: Wiley-Interscience.

    • Google Scholar
12. 1. Curto C
    2. Itskov V
    3. Morrison K
    4. Roth Z
    5. Walker JL

    (2013) Combinatorial neural codes from a mathematical coding theory perspective

    Neural Computation 25:1891–1925.

    https://doi.org/10.1162/NECO_a_00459

    • Google Scholar
13. 1. Dan Y
    2. Alonso JM
    3. Usrey WM
    4. Reid RC

    (1998) Coding of visual information by precisely correlated spikes in the lateral geniculate nucleus

    Nature Neuroscience 1:501–507.

    https://doi.org/10.1038/2217

    • Google Scholar
14. 1. de la Rocha J
    2. Doiron B
    3. Shea-Brown E
    4. Josić K
    5. Reyes A

    (2007) Correlation between neural spike trains increases with firing rate

    Nature 448:802–806.

    https://doi.org/10.1038/nature06028

    • Google Scholar
15. 1. de Ruyter van Steveninck RR
    2. Lewen GD
    3. Strong SP
    4. Koberle R
    5. Bialek W

    (1997) Reproducibility and variability in neural spike trains

    Science 275:1805–1808.

    https://doi.org/10.1126/science.275.5307.1805

    • Google Scholar
16. 1. Diesmann M
    2. Gewaltig MO
    3. Aertsen A

    (1999) Stable propagation of synchronous spiking in cortical neural networks

    Nature 402:529–533.

    https://doi.org/10.1038/990101

    • Google Scholar
17. 1. Ecker AS
    2. Berens P
    3. Keliris GA
    4. Bethge M
    5. Logothetis NK
    6. Tolias AS

    (2010) Decorrelated neuronal firing in cortical microcircuits

    Science 327:584–587.

    https://doi.org/10.1126/science.1179867

    • Google Scholar
18. 1. Faisal AA
    2. Selen LP
    3. Wolpert DM

    (2008) Noise in the nervous system

    Nature Reviews Neuroscience 9:292–303.

    https://doi.org/10.1038/nrn2258

    • Google Scholar
19. 1. Fujisawa S
    2. Amarasingham A
    3. Harrison MT
    4. Buzsaki G

    (2008) Behavior-dependent short-term assembly dynamics in the medial prefrontal cortex

    Nature Neuroscience 11:823–833.

    https://doi.org/10.1038/nn.2134

    • Google Scholar
20. 1. Ganmor E
    2. Segev R
    3. Schneidman E

    (2011a) Sparse low-order interaction network underlies a highly correlated and learnable neural population code

    Proceedings of the National Academy of Sciences of USA 108:9679–9684.

    https://doi.org/10.1073/pnas.1019641108

    • Google Scholar
21. 1. Ganmor E
    2. Segev R
    3. Schneidman E

    (2011b) The architecture of functional interaction networks in the retina

    The Journal of Neuroscience 31:3044–3054.

    https://doi.org/10.1523/JNEUROSCI.3682-10.2011

    • Google Scholar
22. 1. Granot-Atedgi E
    2. Tkačik G
    3. Segev R
    4. Schneidman E

    (2013) Stimulus-dependent maximum entropy models of neural population codes

    PLOS Computational Biology 9:e1002922.

    https://doi.org/10.1371/journal.pcbi.1002922

    • Google Scholar
23. 1. Harvey CD
    2. Coen P
    3. Tank DW

    (2012) Choice-specific sequences in parietal cortex during a virtual-navigation decision task

    Nature 484:62–68.

    https://doi.org/10.1038/nature10918

    • Google Scholar
24. 1. Haslinger R
    2. Pipa G
    3. Lewis LD
    4. Nikolić D
    5. Williams Z
    6. Brown E

    (2013) Encoding through patterns: Regression tree–based neuronal population models

    Neural Computation 25:1953–1993.

    https://doi.org/10.1162/NECO_a_00464

    • Google Scholar
25. 1. Hopfield JJ

    (1982) Neural networks and physical systems with emergent collective computational abilities

    Proceedings of the National Academy of Sciences of USA 79:2554–2558.

    https://doi.org/10.1073/pnas.79.8.2554

    • Google Scholar
26. 1. Houghton C
    2. Sen K

    (2008) A new multineuron spike train metric

    Neural Computation 20:1495–1511.

    https://doi.org/10.1162/neco.2007.10-06-350

    • Google Scholar
27. 1. Hubert L
    2. Schultz J

    (2011) Quadratic assignment as a general data analysis strategy

    The British Journal of Mathematical and Statistical Psychology 29:190–241.

    https://doi.org/10.1111/j.2044-8317.1976.tb00714.x

    • Google Scholar
28. 1. Huth AG
    2. Nishimoto S
    3. Vu AT
    4. Gallant JL

    (2012) A continuous semantic space describes the representation of thousands of object and action categories across the human brain

    Neuron 76:1210–1224.

    https://doi.org/10.1016/j.neuron.2012.10.014

    • Google Scholar
29. 1. Jaynes ET

    (1957) Information theory and statistical Mechanics

    Physical Review 106:620–630.

    https://doi.org/10.1103/PhysRev.106.620

    • Google Scholar
30. 1. Kohn A
    2. Smith MA

    (2005) Stimulus dependence of neuronal correlation in primary visual cortex of the Macaque

    The Journal of Neuroscience 25:3661–3673.

    https://doi.org/10.1523/JNEUROSCI.5106-04.2005

    • Google Scholar
31. 1. Loebel A
    2. Nelken I
    3. Tsodyks M

    (2007) Processing of sounds by population spikes in a model of primary auditory cortex

    Frontier in Neuroscience 1:197–209.

    https://doi.org/10.3389/neuro.01.1.1.015.2007

    • Google Scholar
32. 1. Mainen ZF
    2. Sejnowski TJ

    (1995) Reliability of spike timing in neocortical neurons

    Science 268:1503–1506.

    https://doi.org/10.1126/science.7770778

    • Google Scholar
33. 1. Maynard EM
    2. Hatsopoulos NG
    3. Ojakangas CL
    4. Acuna BD
    5. Sanes JN
    6. Normann RA
    7. Donoghue JP

    (1999)

    Neuronal interactions improve cortical population coding of movement direction

    The Journal of Neuroscience 19:8083–8093.

    • Google Scholar
34. 1. Mazor O
    2. Laurent G

    (2005) Transient dynamics versus fixed points in odor representations by Locust antennal lobe projection neurons

    Neuron 48:661–673.

    https://doi.org/10.1016/j.neuron.2005.09.032

    • Google Scholar
35. 1. Meister M
    2. Pine J
    3. Baylor DA

    (1994) Multi-neuronal signals from the retina: acquisition and analysis

    Journal of Neuroscience Methods 51:95–106.

    https://doi.org/10.1016/0165-0270(94)90030-2

    • Google Scholar
36. 1. Nicolelis MA
    2. Baccala LA
    3. Lin RC
    4. Chapin JK

    (1995) Sensorimotor encoding by synchronous neural ensemble activity at multiple levels of the somatosensory system

    Science 268:1353–1358.

    https://doi.org/10.1126/science.7761855

    • Google Scholar
37. 1. Nirenberg S
    2. Carcieri SM
    3. Jacobs AL
    4. Latham PE

    (2001) Retinal ganglion cells act largely as independent encoders

    Nature 411:698–701.

    https://doi.org/10.1038/35079612

    • Google Scholar
38. 1. Ohiorhenuan IE
    2. Mechler F
    3. Purpura KP
    4. Schmid AM
    5. Hu Q
    6. Victor JD

    (2010) Sparse coding and high-order correlations in fine-scale cortical networks

    Nature 466:617–621.

    https://doi.org/10.1038/nature09178

    • Google Scholar
39. 1. Oizumi M
    2. Ishii T
    3. Ishibashi K
    4. Hosoya T
    5. Okada M

    (2010) Mismatched decoding in the brain

    The Journal of Neuroscience 30:4815–4826.

    https://doi.org/10.1523/JNEUROSCI.4360-09.2010

    • Google Scholar
40. 1. Osborne LC
    2. Lisberger SG
    3. Bialek W

    (2005) A sensory source for motor variation

    Nature 437:412–416.

    https://doi.org/10.1038/nature03961

    • Google Scholar
41. 1. Parnas M
    2. Lin AC
    3. Huetteroth W
    4. Miesenböck G

    (2013) Odor discrimination in Drosophila: from neural population codes to behavior

    Neuron 79:932–944.

    https://doi.org/10.1016/j.neuron.2013.08.006

    • Google Scholar
42. Book

    1. Pereira F
    2. Tishby N
    3. Lee L

    (1993)

    Distributional clustering of english words, in: proceedings of the 31st annual meeting on association for computational linguistics, ACL '93

    Stroudsburg, PA, USA: Association for Computational Linguistics.

    • Google Scholar
43. 1. Pillow JW
    2. Shlens J
    3. Paninski L
    4. Sher A
    5. Litke AM
    6. Chichilnisky EJ
    7. Simoncelli EP

    (2008) Spatio-temporal correlations and visual signalling in a complete neuronal population

    Nature 454:995–999.

    https://doi.org/10.1038/nature07140

    • Google Scholar
44. 1. Pola G
    2. Thiele A
    3. Hoffmann KP
    4. Panzeri S

    (2003) An exact method to quantify the information transmitted by different mechanisms of correlational coding

    Network 14:35–60.

    https://doi.org/10.1088/0954-898X/14/1/303

    • Google Scholar
45. 1. Puchalla JL
    2. Schneidman E
    3. Harris RA
    4. Berry MJ

    (2005) Redundancy in the population code of the retina

    Neuron 46:493–504.

    https://doi.org/10.1016/j.neuron.2005.03.026

    • Google Scholar
46. 1. Reich DS
    2. Mechler F
    3. Victor JD

    (2001) Independent and redundant information in nearby cortical neurons

    Science 294:2566–2568.

    https://doi.org/10.1126/science.1065839

    • Google Scholar
47. Book

    1. Rieke F
    2. Warland D
    3. de Ruyter van Steveninck R
    4. Bialek W

    (1999)

    Spikes: exploring the neural code

    Cambridge, MA: The MIT Press.

    • Google Scholar
48. 1. Rolls ET
    2. Treves A

    (2011) The neuronal encoding of information in the brain

    Progress in Neurobiology 95:448–490.

    https://doi.org/10.1016/j.pneurobio.2011.08.002

    • Google Scholar
49. 1. Schneidman E
    2. Berry MJ II
    3. Segev R
    4. Bialek W

    (2006) Weak pairwise correlations imply strongly correlated network states in a neural population

    Nature 440:1007–1012.

    https://doi.org/10.1038/nature04701

    • Google Scholar
50. 1. Schneidman E
    2. Bialek W
    3. Berry MJ II

    (2003a)

    Synergy, redundancy, and independence in population codes

    The Journal of Neuroscience 23:11539–11553.

    • Google Scholar
51. 1. Schneidman E
    2. Freedman B
    3. Segev I

    (1998) Ion channel stochasticity may be critical in determining the reliability and precision of spike timing

    Neural Computation 10:1679–1703.

    https://doi.org/10.1162/089976698300017089

    • Google Scholar
52. 1. Schneidman E
    2. Still S
    3. Berry MJ II
    4. Bialek W

    (2003b) Network information and connected correlations

    Physical Review Letters 91:238701.

    https://doi.org/10.1103/PhysRevLett.91.238701

    • Google Scholar
53. 1. Schwartz G
    2. Macke J
    3. Amodei D
    4. Tang H
    5. Berry MJ II

    (2012) Low error discrimination using a correlated population code

    Journal of Neurophysiology 108:1069–1088.

    https://doi.org/10.1152/jn.00564.2011

    • Google Scholar
54. 1. Shlens J
    2. Field GD
    3. Gauthier JL
    4. Greschner M
    5. Sher A
    6. Litke AM
    7. Chichilnisky EJ

    (2009) The structure of large-scale synchronized firing in primate retina

    The Journal of Neuroscience 29:5022–5031.

    https://doi.org/10.1523/jneurosci.5187-08.2009

    • Google Scholar
55. Book

    1. Slonim N
    2. Tishby N

    (1999)

    Agglomerative information bottleneck

    MIT Press.

    • Google Scholar
56. 1. Sompolinsky H
    2. Yoon H
    3. Kang K
    4. Shamir M

    (2001) Population coding in neuronal systems with correlated noise

    Physical Review. E, Statistical, Nonlinear, and Soft Matter Physics 64:051904.

    https://doi.org/10.1103/PhysRevE.64.051904

    • Google Scholar
57. 1. Sreenivasan S
    2. Fiete I

    (2011) Grid cells generate an analog error-correcting code for singularly precise neural computation

    Nature Neuroscience 14:1330–1337.

    https://doi.org/10.1038/nn.2901

    • Google Scholar
58. 1. Stocker AA
    2. Simoncelli EP

    (2006) Noise characteristics and prior expectations in human visual speed perception

    Nature Neuroscience 9:578–585.

    https://doi.org/10.1038/nn1669

    • Google Scholar
59. 1. Strong SP
    2. Koberle R
    3. de Ruyter van Steveninck RR
    4. Bialek W

    (1998) Entropy and information in neural spike trains

    Physical Review Letters 80:197.

    https://doi.org/10.1103/PhysRevLett.80.197

    • Google Scholar
60. 1. Tenenbaum JB
    2. de Silva V
    3. Langford JC

    (2000) A global geometric framework for nonlinear dimensionality reduction

    Science 290:2319–2323.

    https://doi.org/10.1126/science.290.5500.2319

    • Google Scholar
61. 1. Tkačik G
    2. Granot-Atedgi E
    3. Segev R
    4. Schneidman E

    (2013) Retinal metric: a stimulus distance measure derived from population neural responses

    Physical Review Letters 110:058104.

    https://doi.org/10.1103/PhysRevLett.110.058104

    • Google Scholar
62. 1. Tkacik G
    2. Schneidman E
    3. Berry MJ II
    4. Bialek W

    (2006)

    Ising models for networks of real neurons

    ArXiv q-Bio0611072.

    • Google Scholar
63. 1. Treves A
    2. Panzeri S

    (1995) The upward bias in measures of information derived from limited data samples

    Neural Computation 7:399–407.

    https://doi.org/10.1162/neco.1995.7.2.399

    • Google Scholar
64. 1. Truccolo W
    2. Hochberg LR
    3. Donoghue JP

    (2010) Collective dynamics in human and monkey sensorimotor cortex: predicting single neuron spikes

    Nature Neuroscience 13:105–111.

    https://doi.org/10.1038/nn.2455

    • Google Scholar
65. 1. van Rossum MC

    (2001) A novel spike distance

    Neural Computation 13:751–763.

    https://doi.org/10.1162/089976601300014321

    • Google Scholar
66. 1. Vargas-Irwin CE
    2. Shakhnarovich G
    3. Yadollahpour P
    4. Mislow JM
    5. Black MJ
    6. Donoghue JP

    (2010) Decoding complete reach and grasp actions from local primary motor cortex populations

    The Journal of Neuroscience 30:9659–9669.

    https://doi.org/10.1523/JNEUROSCI.5443-09.2010

    • Google Scholar
67. 1. Victor JD

    (2005) Spike train metrics

    Current Opinion in Neurobiology 15:585–592.

    https://doi.org/10.1016/j.conb.2005.08.002

    • Google Scholar
68. 1. Victor JD
    2. Purpura KP

    (1997) Metric-space analysis of spike trains: theory, algorithms and application

    Network Computation in Neural Systems 8:127–164.

    https://doi.org/10.1088/0954-898X_8_2_003

    • Google Scholar
69. 1. Vidne M
    2. Ahmadian Y
    3. Shlens J
    4. Pillow JW
    5. Kulkarni J
    6. Litke AM
    7. Chichilnisky EJ
    8. Simoncelli E
    9. Paninski L

    (2012) Modeling the impact of common noise inputs on the network activity of retinal ganglion cells

    Journal of Computational Neuroscience 33:97–121.

    https://doi.org/10.1007/s10827-011-0376-2

    • Google Scholar
70. 1. Warland DK
    2. Reinagel P
    3. Meister M

    (1997)

    Decoding visual information from a population of retinal ganglion cells

    Journal of Neurophysiology 78:2336–2350.

    • Google Scholar
71. 1. Zemel RS
    2. Dayan P
    3. Pouget A

    (1998) Probabilistic interpretation of population codes

    Neural Computation 10:403–430.

    https://doi.org/10.1162/089976698300017818

    • Google Scholar
72. 1. Zohary E
    2. Shadlen MN
    3. Newsome WT

    (1994) Correlated neuronal discharge rate and its implications for psychophysical performance

    Nature 370:140–143.

    https://doi.org/10.1038/370140a0

    • Google Scholar

Author details

1. Elad Ganmor

   Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel

   Contribution

   EG, Initiated, performed, and wrote the work

   Competing interests

   The authors declare that no competing interests exist.

   "This ORCID iD identifies the author of this article:" 0000-0003-4834-5713

2. Ronen Segev

   Department of Life Sciences, Zlotowski Center for Neuroscience, Ben-Gurion University of the Negev, Beer-Sheva, Israel

   Contribution

   RS, Initiated, performed, and wrote the work

   Competing interests

   The authors declare that no competing interests exist.

3. Elad Schneidman

   Department of Neurobiology, Weizmann Institute of Science, Rehovot, Israel

   Contribution

   ES, Initiated, performed, and wrote the work

   For correspondence

   elad.schneidman@weizmann.ac.il

   Competing interests

   The authors declare that no competing interests exist.

• 5,445

  views

• 947

  downloads

• 48

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.