Neural assemblies uncovered by generative modeling explain wholebrain activity statistics and reflect structural connectivity
Abstract
Patterns of endogenous activity in the brain reflect a stochastic exploration of the neuronal state space that is constrained by the underlying assembly organization of neurons. Yet, it remains to be shown that this interplay between neurons and their assembly dynamics indeed suffices to generate wholebrain data statistics. Here, we recorded the activity from ∼40,000 neurons simultaneously in zebrafish larvae, and show that a datadriven generative model of neuronassembly interactions can accurately reproduce the mean activity and pairwise correlation statistics of their spontaneous activity. This model, the compositional Restricted Boltzmann Machine (cRBM), unveils ∼200 neural assemblies, which compose neurophysiological circuits and whose various combinations form successive brain states. We then performed in silico perturbation experiments to determine the interregional functional connectivity, which is conserved across individual animals and correlates well with structural connectivity. Our results showcase how cRBMs can capture the coarsegrained organization of the zebrafish brain. Notably, this generative model can readily be deployed to parse neural data obtained by other largescale recording techniques.
Editor's evaluation
Large scale recordings, sometimes involving 10s of thousands of neurons, are becoming increasingly common. Making sense of these recordings, however, is not easy. This paper introduces a new method, the compositional Restricted Boltzmann Machine, that overcomes this problem  it can find structure in data, including both "cell assemblies" and structural connectivity, without inordinate computing resources (data from 40,000 neurons recorded from zebrafish can be analyzed in less than a day). This is a valuable contribution, both to those interested in data analysis, and to those interested in zebrafish.
https://doi.org/10.7554/eLife.83139.sa0Introduction
The brain is a highly connected network, organized across multiple scales, from local circuits involving just a few neurons to extended networks spanning multiple brain regions (White et al., 1986; Song et al., 2005; Kunst et al., 2019). Concurrent with this spatial organization, brain activity exhibits correlated firing among large groups of neurons, often referred to as neural assemblies (Harris, 2005). This assembly organization of brain dynamics has been observed in, for example, auditory cortex (Bathellier et al., 2012), motor cortex (Narayanan et al., 2005), prefrontal cortex (Tavoni et al., 2017), hippocampus (Lin et al., 2005), retina (Shlens et al., 2009), and zebrafish optic tectum (Romano et al., 2015; Mölter et al., 2018; Diana et al., 2019; Triplett et al., 2020). These neural assemblies are thought to form elementary computational units and subserve essential cognitive functions such as shortterm memory, sensorimotor computation or decisionmaking (Hebb, 1949; Gerstein et al., 1989; Harris, 2005; Buzsáki, 2010; Harris, 2012; Palm et al., 2014; Eichenbaum, 2018). Despite the prevalence of these assemblies across the nervous system and their role in neural computation, it remains an open challenge to extract the assembly organization of a full brain and to show that the assembly activity state, derived from that of the neurons, is sufficient to account for the collective neural dynamics.
The need to address this challenge is catalyzed by technological advances in lightsheet microscopy, enabling the simultaneous recording of the majority of neurons in the zebrafish brain at singlecell resolution in vivo (Panier et al., 2013; Ahrens et al., 2013; Wolf et al., 2015; Wolf et al., 2017; Migault et al., 2018; Vanwalleghem et al., 2018). This neural recording technique opens up new avenues for constructing nearcomplete models of neural activity, and in particular its assembly organization. Recent attempts have been made to identify assemblies using either clustering (Panier et al., 2013; Triplett et al., 2018; Chen et al., 2018; Mölter et al., 2018; Bartoszek et al., 2021), dimensionality reduction approaches (LopesdosSantos et al., 2013; Romano et al., 2015; Mu et al., 2019) or latent variable models (Diana et al., 2019; Triplett et al., 2020), albeit often limited to single brain regions. However, these methods do not explicitly assess to what extent the inferred assemblies could give rise to the observed neural data statistics, which is a crucial property of physiologically meaningful assemblies (Harris, 2005). Here, we address this challenge by developing a generative model of neural activity that is explicitly constrained by the assembly organization, thereby quantifying if assemblies indeed suffice to produce the observed neural data statistics.
Specifically, we formalize neural assemblies using a bipartite network of two connected layers representing the neuronal and the assembly activity, respectively. Together with the maximum entropy principle (Jaynes, 1957; Bialek, 2012), this architecture defines the Restrictive Boltzmann Machine (RBM) model (Hinton and Salakhutdinov, 2006). Here, we use an extension to the classical RBM definition termed compositional RBM (cRBM) that we have recently introduced (Tubiana and Monasson, 2017; Tubiana et al., 2019a) and which brings multiple advances to assemblybased network modeling: (1) The maximum entropy principle ensures that neural assemblies are inferred solely from the data statistics. (2) The generative nature of the model, through alternate data sampling of the neuronal and assembly layers, can be leveraged to evaluate its capacity to replicate the empirical data statistics, such as the pairwise coactivation probabilities of all neuron pairs. (3) The cRBM steers the assembly organization to the socalled compositional phase where a small number of assemblies are active at any point in time, making the resulting model highly interpretable as we have shown previously for protein sequence analysis (Tubiana et al., 2019b).
Here, we have successfully trained cRBMs to brainscale, neuronlevel recordings of spontaneous activity in larval zebrafish containing 41,000 neurons on average (Panier et al., 2013; Wolf et al., 2017; Migault et al., 2018). This represents an increase of ∼2 orders of magnitude in number of neurons with respect to previously reported RBM implementations (Köster et al., 2014; Gardella et al et al., 2017; Volpi et al., 2020), attained through significant algorithmic and computational enhancements. We found that all cells could be grouped into 100–200 partially overlapping assemblies, which are anatomically localized and together span the entire brain, and accurately replicate the first and second order statistics of the neural activity. These assemblies were found to carry more predictive power than a fully connected model which has orders of magnitude more parameters, validating that assemblies underpin collective neural dynamics. Further, the probabilistic nature of our model allowed us to compute a functional connectivity matrix by quantifying the effect of activity perturbations in silico. This assemblybased functional connectivity is wellconserved across individual fish and consistent with anatomical connectivity at the mesoscale (Kunst et al., 2019).
In summary, we present an assembly decomposition spanning the zebrafish brain, which accurately accounts for its activity statistics. Our cRBM model provides a widely applicable tool to the community to construct lowdimensional data representations that are defined by the statistics of the data, in particular for very highdimensional systems. Its generative capability further allows to produce new (synthetic) activity patterns that are amenable to direct in silico perturbation and ablation studies.
Results
Compositional RBMs construct Hidden Units by grouping neurons into assemblies
Spontaneous neural activity was recorded from eight zebrafish larvae aged 5–7 days post fertilization expressing the GCaMP6s or GCaMP6f calcium reporters using lightsheet microscopy (Panier et al., 2013; Wolf et al., 2017; Migault et al., 2018). Each data set contained the activity of a large fraction of the neurons in the brain ($40709\pm 13854$; mean ± standard deviation), which, after cell segmentation, were registered onto the ZBrain atlas (Randlett et al., 2015) and mapzebrain atlas (Kunst et al., 2019). Individual neuronal fluorescence traces were deconvolved to binarized spike trains using blind sparse deconvolution (Tubiana et al., 2020). This data acquisition process is depicted in Figure 1A.
We trained compositional Restricted Boltzmann Machine (cRBM) models to capture the activity statistics of these neural recordings. cRBMs are maximum entropy models, that is, the maximally unconstrained solution that fits modelspecific data statistics (Hinton and Salakhutdinov, 2006; Tubiana and Monasson, 2017; Gardella et al., 2019), and critically extend the classical RBM formulation. Its architecture consists of a bipartite graph where the highdimensional layer of neurons $\mathbf{\text{v}}$ (named ‘visible units’ in RBM terminology) is connected to the lowdimensional layer of latent components, termed Hidden Units (HUs) $\mathbf{\text{h}}$. Their interaction is characterized by a weight matrix $\mathbf{\text{W}}$ that is regularized to be sparse. The collection of neurons that have nonzero interactions with a particular HU, noted ${h}_{\mu}$ (i.e. with ${w}_{i,\mu}>0$), define its corresponding neural assembly μ (Figure 1B). This weight matrix, together with the neuron weight vector $\mathbf{\text{g}}$ and HU potential $\mathcal{U}$, defines the transformation from the binarized neural activity $\text{v}(t)$ to the continuous HU activity $\text{h}(t)$ (Figure 1B). Figure 1C shows all recorded neurons of a zebrafish brain, colorlabeled according to their strongestconnecting HU, illustrating that cRBMinferred assemblies (hereafter, neural assemblies for conciseness) are typically densely localized in space and together span the entire brain.
Beyond its architecture (Figure 2A), the model is defined by the probability function $P(\text{v},\text{h})$ of any data configuration $(\text{v},\text{h})$ (see Materials and methods ‘Restricted Boltzmann Machines’ and ‘Compositional Restricted Boltzmann Machine’ for details):
where $Z$ is the partition function that normalizes Equation 1 and $E$ is the following energy function:
HU activity h is obtained by sampling from the conditional probability function $P(\text{h}\text{v})$:
Conversely, neural activity is obtained from HU activity through:
Equations 3 and 4 mathematically reflect the dual relationship between neural and assembly states: the Hidden Units $\mathbf{\text{h}}$ drive ‘visible’ neural activity $\mathbf{\text{v}}$, expressed as $P(\text{v}\text{h})$, while the stochastic assembly activity $\mathbf{\text{h}}$ itself is defined as a function of the activity of the neurons: $P(\text{h}\text{v})$. Importantly, the model does not include direct connections between neurons, hence neural correlations $\u27e8{v}_{i}{v}_{j}\u27e9$ can arise solely from shared assemblies. Moreover, this bipartite architecture ensures that the conditional distributions factorize, leading to a sampling procedure where all neurons or all HUs can be sampled in parallel. The cRBM leverages this property to efficiently generate new data by Monte Carlo sampling alternately from $P(\text{h}\text{v})$ and $P(\text{v}\text{h})$ (Figure 2B).
The cRBM differs from the classical RBM formulation (Hinton and Salakhutdinov, 2006) through the introduction of double Rectified Linear Unit (dReLU) potentials $\mathcal{U}}_{\mu$, weight sparsity regularization and normalized HU activity (further detailed in Methods). We have previously demonstrated in theory and application (Tubiana and Monasson, 2017; Tubiana et al., 2019a; Tubiana et al., 2019b) that this new formulation steers the model into the socalled compositional phase, which makes the latent representation highly interpretable. This phase occurs when a limited number $m$ of HUs coactivate such that $1\ll m\ll M$ where $M$ is the total number of HUs. Thus, each visible configuration is mapped to a specific combination of activated HUs. This contrasts with the ferromagnetic phase ($m\sim 1$) where each HU encodes one specific activity pattern, thus severely limiting the possible number of encoded patterns, or the spinglass phase ($m\sim M$) where all HUs activate simultaneously, yielding a very complex assembly patchwork (Tubiana and Monasson, 2017). Therefore, the compositional phase can provide the right level of granularity for a meaningful interpretation of the cRBM neural assemblies by decomposing the overall activity as a timedependent coactivation of different assemblies of interpretable size and extent.
Trained cRBMs accurately replicate data statistics
cRBM models are trained to maximize the $P(\text{v},\text{h})$ loglikelihood of the zebrafish data recordings, which is achieved by matching the modelgenerated statistics $\u27e8{v}_{i}\u27e9$, $\u27e8{h}_{\mu}\u27e9$ and $\u27e8{v}_{i}{h}_{\mu}\u27e9$ (the mean neuronal activity, mean HU activity and their correlations, respectively) to the empirical data statistics (Equation 14). In order to optimize the two free parameters of the cRBM model – the sparsity regularization parameter $\lambda $ and the total number of HUs $M$ – we assessed the cRBM performance for a grid of $(\lambda ,M)$values for one data set (fish #3). This analysis yielded an optimum for $\lambda =0.02$ and $M=200$ (Figure 2—figure supplement 1). These values were subsequently used for all recordings, where $M$ was scaled with the number of neurons $N$.
We trained cRBMs on 70% of the recording length, and compared the statistics of modelgenerated data to the withheld test data set (the remaining 30% of recording, see Materials and methods ‘Train / test data split’ and ‘Assessment of data statistics’ for details). After convergence, the cRBM generated data that replicated the training statistics accurately, with normalized Root Mean Square Error (nRMSE) values of ${\text{nRMSE}}_{\u27e8{v}_{i}\u27e9}=0.11$, ${\text{nRMSE}}_{\u27e8{h}_{\mu}\u27e9}=0.15$ and ${\text{nRMSE}}_{\u27e8{v}_{i}{h}_{\mu}\u27e9}=0.09$ (Figure 2CE). Here, nRMSE is normalized such that 1 corresponds to shuffled data statistics and 0 corresponds to the best possible RMSE, i.e., between train and test data.
We further evaluated cRBM performance to assess its ability to capture data statistics that the cRBM was not explicitly trained to replicate: the pairwise correlations between neurons $\u27e8{v}_{i}{v}_{j}\u27e9$ and the pairwise correlations between HUs $\u27e8{h}_{\mu}{h}_{\nu}\u27e9$. We found that these statistics were also accurately replicated by modelgenerated data, with ${\text{nRMSE}}_{\u27e8{v}_{i}{v}_{j}\u27e9}=0.09$ (meaning that the model slightly outperformed the traintest data difference) and ${\text{nRMSE}}_{\u27e8{h}_{\mu}{h}_{\nu}\u27e9}=0.17$ (Figure 2F, G). The fact that cRBM also accurately replicated neural correlations $\u27e8{v}_{i}{v}_{j}\u27e9$ (Figure 2F) is of particular relevance, since this indicates that (1) the assumption that neural correlations can be explained by their shared assemblies is justified and (2) cRBMs may provide an efficient mean to model neural interactions of such large systems ($N\sim {10}^{4}$) where directly modeling all ${N}^{2}$ interactions would be computationally infeasible or not sufficiently constrained by the available data.
Next, we assessed the reconstruction quality after neural data was compressed by the cRBM lowdimensional bottleneck. The reconstruction quality is defined as the loglikelihood of reconstructed neural data ${\text{v}}_{\text{recon}}$ (i.e. $\mathbf{\text{v}}$ that is first transformed to the lowdimensional $\mathbf{\text{h}}$, and then back again to the highdimensional ${\text{v}}_{\text{recon}}$, see Materials and methods  ‘Reconstruction quality’). This is important to prevent trivial, undesired solutions like ${w}_{i,\mu}=0\forall i,\mu $ which would directly lead to ${\u27e8{h}_{\mu}\u27e9}_{P(\text{v},\text{h})}={\u27e8{h}_{\mu}\u27e9}_{\text{data}}=0$ (potentially because of strong sparsity regularization). Figure 2H shows the distribution of cRBM reconstruction quality of all neurons (in purple), quantified by the normalized loglikelihood (nLLH) such that 0 corresponds to an independent model $(P({v}_{i}(t))=\u27e8{v}_{i}\u27e9)$ and 1 corresponds to perfect reconstruction (nonnormalized $\text{LLH}=0$). For comparison, we also reconstructed the neural activity using a fully connected Generalized Linear Model (GLM, see Materials and methods  ‘Generalized Linear Model’ and Figure 2, Figure 2—figure supplement 2H, blue). The cRBM nLLH distribution is significantly greater than the GLM nLLH distribution (onesided Mann Whitney U test, $P<{10}^{42}$), with medians ${\text{LLH}}_{\text{cRBM}}=0.24$ and ${\text{LLH}}_{\text{GLM}}=0.20$. Hence, projecting the neural data onto the lowdimensional representation of the HUs does not compromise the ability to explain the neural activity. In fact, reconstruction quality of the cRBM slightly outperforms the GLM, possibly due to the suppression of noise in the cRBM estimate. The optimal $(\lambda =0.02,M=200)$ choice of free parameters was selected by crossvalidating the median of the cRBM reconstruction quality, together with the normalized RMSE of the five previously described statistics (Figure 2—figure supplement 1).
Lastly, we confirmed that the cRBM indeed resides in the compositional phase, characterized by $1\ll m(t)\ll M$ where $m(t)$ is the number of HUs active at time point $t$ (Figure 2—figure supplement 3A). This property is a consequence of the sparse weight matrix $\mathbf{\text{W}}$, indicated by its heavytail logdistribution (Figure 2I, purple). The compositional phase is the norm for the presently estimated cRBMs, evidenced by the distribution of median $m(t)$ values for all recordings (average $\frac{\mathrm{median}(m)}{M}$ is 0.26, see Figure 2—figure supplement 3B). Importantly, the sparse weight matrix does not automatically imply that only a small subset of neurons is connected to the cRBM hidden layer. We validated this by observing that more neurons strongly connect to the hidden layer than expected by shuffling the weight matrix (Figure 2J). Further, we quantified the number of assemblies that each neuron was embedded in, which showed that increasing the embedding threshold did not notably affect the fraction of neurons embedded in at least 1 assembly (93–94%, see Figure 2—figure supplement 4). To assess the influence of $M$ and $\lambda $ on the inferred assemblies, we computed, for all cRBM models trained during the optimization of $M$ and $\lambda $, the distribution of assembly sizes (Figure 2—figure supplement 5AF). We found that $M$ and $\lambda $ controlled the distribution of assembly sizes in a consistent manner: assembly size was a gradually decreasing function of both $M$ and $\lambda $ (twoway ANOVA, both $P<{10}^{3}$). Furthermore, for $M$ and $\lambda $ values close to the optimal parametersetting ($M=200$, $\lambda =0.02$), the changes in assembly size were very small and gradual. This showcases the robustness of the cRBM to slight changes in parameter choice.
Sparsity facilitated that each assembly only connects to a handful of anatomical regions, as we quantified by calculating the overlap between cRBM assemblies and anatomical regions (Figure 2—figure supplement 6). We found that cRBM assemblies connect to a median of three regions (interquartile range: 2–6 regions). Importantly, the cRBM has no information about the locations of neurons during training, so the localization to a limited set of anatomical areas that we observe is extracted from the neural coactivation properties alone. For comparison, Principal Component Analysis (PCA), a commonly used nonsparse dimensionality reduction method that shares the cRBM architecture, naturally converged to a nonsparse weight matrix (Figure 2I, yellow), with fewer connected neurons than expected by shuffling its weight matrix (Figure 2K). This led to unspecific assemblies that are difficult to interpret by anatomy (Figure 2—figure supplement 6). As a result, sparsity, a cRBM property shared with some other dimensionality reduction techniques, is crucial to interpret the assemblies by anatomy as we demonstrate in the next section.
We next asked whether sparsity alone was sufficient for a generative model to accurately recapitulate the neural recording statistics. To address this question, we trained sparse linear Variational Autoencoders (VAEs) using the same parameteroptimization protocol (Figure 2—figure supplement 7A). Like cRBMs, linear VAEs are generative models that learn a latent representation of a dataset (Tubiana et al., 2019a). We observed that VAEs were not able to replicate the secondorder statistics, and therefore were not able to reconstruct neural activity from latent representation (Figure 2—figure supplement 7BD), even though they also obtained sparse representations (Figure 2—figure supplement 7E, F). Other clustering or dimensionality reduction methods, such as kmeans, PCA and nonnegative matrix factorization, have been used previously to cluster neurons in the zebrafish brain (Chen et al., 2018; Mu et al., 2019; Marques et al., 2020). However, because these methods cannot generate artificial neural data using their inferred assemblies, their quality cannot be quantitatively assessed as we have done for the cRBM (but see Tubiana et al., 2019a for other comparisons).
cRBM assemblies compose functional circuits and anatomical structures
Above, we have shown that cRBMs converge to sparse weight matrix solutions. This property enables us to visualize the cRBMinferred neural assemblies as the collection of significantly connected neurons to an HU. Neurons from a given neural assembly display concerted dynamics, and so one may expect their spatial organization to reflect the neuroanatomy and functional organization of the brain. We here highlight a selection of salient examples of neural assemblies, illustrating that assemblies match well with anatomical structures and functional circuits, while the complete set of neural assemblies is presented in Video 1. In particular, we identified assemblies that together compose a neural circuit, are neurotransmitterspecific, encompass a longrange pathway, or can be identified by anatomy. The examples shown here are from a single fish (#3), but results from other fish were comparable.
First, we identified six assemblies that together span the hindbrain circuit that drives eye and tail movements (Dunn et al., 2016; Wolf et al., 2017; Chen et al., 2018). We find two neural assemblies in rhombomere 2 which align with the anterior rhombencephalic turning region (ARTR, Ahrens et al., 2013; Dunn et al., 2016; Wolf et al., 2017, Figure 3A, B). Each assembly primarily comprises neurons of either the left or right side of the ARTR, but also includes a small subset of contralateral neurons with weights of opposite sign in line with the established mutual inhibition between both subpopulations. Two other symmetric assemblies (Figure 3C, D) together encompass the oculomotor nucleus (nIII) and the contralateral abducens nucleus (nVI, in rhombomere 6), two regions engaged in ocular saccades (Ma et al., 2014) and under the control of the ARTR (Wolf et al., 2017). Additionally, we observed two symmetric assemblies (Figure 3E, F) in the posterior hindbrain (in rhombomere 7), in a region known to drive unilateral tail movements (Chen et al., 2018; Marques et al., 2020) and whose antiphasic activation is also controlled by the ARTR activity (Dunn et al., 2016).
Next, we observed assemblies that correspond to particular neurotransmitter expressions in the ZBrain atlas (Randlett et al., 2015), such as the excitatory Vglut2 (Figure 3G) and inhibitory Gad1b (Figure 3H) neurotransmitters. These assemblies consist of multiple dense loci that sparsely populate the entire brain, confirming that cRBMs are able to capture a large morphological diversity of neural assemblies. Figure 3I depicts another sparse, brainwide assembly that encompasses the pallium, habenula (Hb) and interpeduncular nucleus (IPN), and thus captures the HbIPN pathway that connects to other regions such as the pallium (Beretta et al., 2012; Bartoszek et al., 2021).
Larger nuclei or circuits were often composed of a small number of distinct neural assemblies with some overlap. For example, the cerebellum was decomposed into multiple, bilateral assemblies (Figure 3J) whereas neurons in the torus semicircularis were grouped per brain hemisphere (Figure 3K). As a last example, the optic tectum was composed of a larger set of approximately 18 neural assemblies, which spatially tiled the volume of the optic tectum (Figure 3L). This particular organization is suggestive of spatially localized interactions within the optic tectum, and aligns with the morphology of previously inferred assemblies in this specific region (Romano et al., 2015; Diana et al., 2019; Triplett et al., 2020). However, Figure 3 altogether demonstrates that the typical assembly morphology of the optic tectum identified by our and these previous analyses does not readily generalize to other brain regions, where a large range of different assembly morphologies compose neural circuits.
Overall, the clear alignment of cRBMbased neural assemblies with anatomical regions and circuits suggests that cRBMs are able to identify anatomical structures from dynamical activity alone, which enables them to break down the overall activity into parts that are interpretable by physiologists in the context of previous, more local studies.
HU dynamics cluster into groups and display slower dynamics than neurons
HU activity, defined as the expected value of $P(\text{h}\text{v})$ (Equation 9), exhibits a rich variety of dynamical patterns (Figure 4A). HUs can activate very transiently, slowly modulate their activity, or display periods of active and inactive states of comparable duration. Figure 4B highlights a few HU activity traces that illustrate this diversity of HU dynamics. The top three panels of Figure 4B show the dynamics of the assemblies of Figure 3AF which encompass the ARTR hindbrain circuit that controls saccadic eye movements and directional tail flips. HUs 99 and 161 drive the left and right ARTR and display antiphasic activity with long dwell times of ∼15s, in accordance with previous studies (Ahrens et al., 2013; Dunn et al., 2016; Wolf et al., 2017). HU 102 and 163 correspond to the oculomotor neurons in the nuclei nIII and nVI that together drive the horizontal saccades. Their temporal dynamics are locked to that of the ARTR units in line with the previously identified role of ARTR as a pacemaker for the eye saccades (Wolf et al., 2017). HUs 95 and 135, which drive directional tail flips, display transient activations that only occur when the ipsilateral ARTRassociated HU is active. This is consistent with the previous finding that the ARTR alternating activation pattern sets the orientation of successive tail flips accordingly (Dunn et al., 2016). The fourth panel shows the traces of the brainwide assemblies of Figure 3G, I, displaying slow tonic modulation of their activity. Finally, the bottom panel, which corresponds to the collective dynamics of assembly 122 (Figure 3H), comprises short transient activity that likely corresponds to fictive swimming events.
Some HUs regularly coactivate, leading to strong correlations between different HUs. This is quantified by their Pearson correlation matrix shown in Figure 4C (top), which reveals clusters of correlated HUs. These were grouped using hierarchical clustering (Figure 4C, bottom), and we then manually identified their main anatomical location (top labels). These clusters of HUs with strongly correlated activity suggest that much of the HU variance could be captured using only a small number of variables. We quantified this by performing PCA on the HU dynamics, finding that indeed 52% of the variance was captured by the first three PCs, and 85% by the first 20 PCs (Figure 4D). We further observed that HU activity is bimodal, as evidenced by the distribution of all HU activity traces in Figure 4E. This bimodality can emerge because the dReLU potentials $\mathcal{U}}_{\mu$ (Equation 13) can learn to take different shapes, including a doublewell potential that leads to bimodal dynamics (see Materials and methods  ‘Choice of HU potential’). This allows us to effectively describe HU activity as a twostate system, where ${h}_{\mu}(t)>0$ increases the probability to spike ($P({v}_{i}(t)=1)$) for its positively connected neurons, and ${h}_{\mu}(t)<0$ decreases their probability to spike. The binarized neuron activity is also a twostate system (spiking or not spiking), which enabled us to compare the time constants of neuron and HU state changes, quantified by the median time between successive onsets of activity. We find that HUs, which represent the concerted dynamics of neuronal assemblies, operate on a slower time scale than individual neurons (Figure 4F, Figure 2—figure supplement 5GL). This observation aligns with the expected difference between cellular and circuitlevel time scales.
cRBM embodies functional connectivity that is strongly correlated across individuals
The probabilistic nature of cRBMs uniquely enables in silico perturbation experiments to estimate the functional connection ${J}_{ij}$ between pairs of neurons, where ${J}_{ij}$ is quantified by directly perturbing the activity of neuron $j$ and observing the change in probability to spike of neuron $i$. We first defined the generic, symmetric functional connection ${J}_{ij}$ using $P({v}_{i}{v}_{j},{v}_{k\ne i,j})$ (Equation 15) and then used $P(\text{v})$ (Equation 12) to derive the cRBMspecific ${J}_{ij}$ (Equation 17, see Materials and methods  ‘Effective connectivity matrix’). Using this definition of ${J}_{ij}$, we constructed a full neurontoneuron effective connectivity matrix for each zebrafish recording. We then asked whether this cRBMinferred connectivity matrix was robust across individuals. For this purpose, we calculated the functional connections between anatomical regions, given by the assemblies that occupy each region, because neuronal identities can vary across individual specimen. We aggregated neurons using the L_{1} norm for each pair of anatomical regions to determine the functional connection between regions (see Materials and methods  ‘From interneuron to interregion connectivity’). For this purpose, we considered anatomical regions as defined by the mapzebrain atlas (Kunst et al., 2019) for which a regionalscale structural connectivity matrix exists to which we will compare our functional connectivity matrix.
This led to a symmetrical functional connectivity matrix for each animal, three of which are shown in Figure 5AC (where nonimaged regions are left blank, and all eight animals are shown in Figure 5—figure supplement 1). The strength of functional connections is distributed approximately lognormal (Figure 5D), similar to the distribution of structural regiontoregion connections (Kunst et al., 2019). To quantify the similarity between individual fish, we computed the Pearson correlation between each pair of fish. Functional connectivity matrices correlate strongly across individuals, with an average Pearson correlation of 0.69 (Figure 5E and F).
We conclude that similar functional circuits spontaneously activate across individuals, despite the limited duration of neural recordings (∼25 minutes), which can be identified across fish using independently estimated cRBMs. In the next section, we aggregate these individual matrices to a general functional connectivity matrix for comparison with the zebrafish structural connectivity matrix.
cRBMinferred functional connectivity reflects structural connectivity
In the previous section we have determined the functional connections between anatomical regions using the cRBM assembly organization. Although functional connectivity stems from the structural (i.e. biophysical) connections between neurons, it can reflect correlations that arise through indirect network interactions (Bassett and Sporns, 2017; Das and Fiete, 2020). Using recently published structural connectivity data of the zebrafish brain (Kunst et al., 2019), we are now able to quantify the overlap between a structurally defined connectivity matrix and our functional connectivity matrix estimated through neural dynamics. Kunst et al., 2019 determined a zebrafish structural connectivity matrix between 72 anatomical regions using structural imaging data from thousands of individually Green Fluorescent Protein (GFP)labeled neurons from multiple animals. We slightly extended this matrix by using the most recent data, filtering indirect connections and accounting for the resulting sampling bias (Figure 6A, regions that were not imaged in our lightsheet microscopy experiments were excluded). Next, we aggregated the functional connectivity matrices of all our calcium imaging recordings to one grand average functional connectivity matrix (Figure 6B).
For comparison, we also calculated the connectivity matrices defined by either covariance or Pearson correlation (Figure 6—figure supplement 1). The cRBM functional connectivity spans a larger range of values than either of these methods, leading to a more finegrained connectivity matrix akin to the structural connectivity map (Figure 6B). This greater visual resemblance was statistically confirmed by calculating the Spearman correlation between structural and functional connectivity, which is greater for cRBM (${r}_{\text{S}}=0.39$, Figure 6C), than for covariancebased connectivity (${r}_{\text{S}}=0.18$, Figure 6—figure supplement 1 left) or correlationbased connectivity (${r}_{\text{S}}=0.26$, Figure 6—figure supplement 1 right). Hence, using recordings of ∼25 min on average, cRBMs were able to identify functional connections that resemble the anatomical connectivity between brain regions. Strong or weak functional connections are predictive of present or absent structural connections respectively (Figure 6D), and could thus potentially be used for inference in systems where the structural connectivity pattern is unknown.
Discussion
We have developed a cRBM model that accurately replicated the data statistics of brainscale zebrafish recordings, thereby forming neural assemblies that spanned the entire brain. The objective of our study was threefold: first, to show that the cRBM model can be applied to highdimensional data, such as wholebrain recordings, second, to prove that an assemblybased model is sufficient to generate wholebrain neural data statistics, and third, to describe the physiological properties of the assembly organization in the zebrafish brain and use it to create a functional connectivity map. We have shown that, after convergence, the cRBMgenerated data not only replicated the data statistics that it was constrained to fit, but also extrapolated to fit the pairwise correlation statistics of neurons and HUs, leading to a better reconstruction of neural data than a fully connected GLM (Figure 2). These results thereby quantify how neural assemblies play a major role in determining the collective dynamics of the brain. To achieve this, cRBMs formed sparsely localized assemblies that spanned the entire brain, facilitating their biological interpretation (Figures 3 and 4, Figure 2—figure supplement 6). Further, the probabilistic nature of the cRBM model allowed us to create a mesoscale functional connectivity map that was largely conserved across individual fish and correlated well with structural connectivity (Figures 5 and 6).
The maximum entropy principle underlying the cRBM definition has been a popular method for inferring pairwise effective connections between neurons or assemblies of coactivating cells (Schneidman et al., 2006; Tavoni et al., 2017; Ferrari et al., 2017; Meshulam et al., 2017; Posani et al., 2018; Chen et al., 2019). However, its computational cost has limited this pairwise connectivity analysis to typically $\sim {10}^{2}$ neurons. The twolayer cRBM model that we used here alleviates this burden, because the large number of neurontoneuron connections are no longer explicitly optimized, which enables a fast data sampling procedure (Figure 2B). However, we have shown that these connections are still estimated indirectly with high accuracy via the assemblies they connect to (Figure 2F). We have thus shown that the cRBM is able to infer the $\frac{1}{2}{N}^{2}\approx {10}^{9}$ (symmetric) pairwise connections through its assembly structure, a feat that is computationally infeasible for many other methods. By implementing various algorithmic optimizations (Materials and methods  ‘Algorithmic Implementation’), cRBM models converged in approximately 8–12 hr on highend desktop computers (also see Materials and methods  ‘Computational limitations’).
Previously, we have extensively compared cRBM performance to other dimensionality reduction techniques, including Principal Component Analysis (PCA), Independent Component Analysis (ICA), Variational Autoencoders (VAEs) and their sparse variants, using protein sequence data as a benchmark (Tubiana et al., 2019a). Briefly put, we showed that PCA and ICA could not accurately model the system due to their deterministic nature, putting too much emphasis on lowprobability highvariance states, while VAEs were unable to capture all features of data due to the unrealistic assumption of independent, Gaussiandistributed latent variables. In this study, we repeated this comparison with sparse linear VAEs, and reached similar conclusions: VAEs trained using the same protocol as cRBMs failed to reproduce secondorder data statistics and to reconstruct neural activity via the latent layer, while the learnt assemblies were of substantially lower quality (indicated by a large fraction of disconnected HUs, as well as a highly variable assembly size; Figure 2—figure supplement 7). Additionally, while PCA has previously been successful in describing zebrafish neural dynamics in terms of their main covariances modes (Ahrens et al., 2012; Marques et al., 2020), we show here that it is not appropriate for assembly extraction due to the absence of both a compositional and stochastic nature (Figure 2, Figure 2—figure supplement 6). Furthermore, we have shown that the generative component of cRBM models is essential for quantitatively assessing that the assembly organization is sufficient for reproducing neural statistics (Figure 2), moving beyond deterministic clustering analyses such as kmeans (Panier et al., 2013; Chen et al., 2018), similarity graph clustering (Mölter et al., 2018) or nonnegative matrix factorization (Mu et al., 2019) (see Supplementary file 2).
After having quantitatively validated the resultant assemblies, we moved to discussing the biological implications of our findings. Previous studies of the zebrafish optic tectum have identified neural assemblies that were spatially organized into single dense clusters of cells (Romano et al., 2015; Diana et al., 2019; Triplett et al., 2020). We have replicated these findings by observing the distinct organization of ballshaped assemblies in the optic tectum (Figure 3L). However, our data extends to many other anatomical regions in the brain, where we found that assemblies can be much more dispersed, albeit still locally dense, consisting of multiple clusters of neurons (Figure 3). In sum, cRBMinferred cell assemblies display many properties that one expects from physiological cell assemblies: they are anatomically localized, can overlap, encompass functionally identified neuronal circuits and underpin the collective neural dynamics (Harris, 2005; Harris, 2012; Eichenbaum, 2018). Yet, the cRBM bipartite architecture lacks many of the traits of neurophysiological circuits. In particular, cRBMs lack direct neurontoneuron connections, asymmetry in the connectivity weights and a hierarchical organization of functional dependencies beyond one hidden layer. Therefore, to what extent cRBMinferred assemblies identify to neurophysiological cell assemblies, as postulated by Hebb, 1949 and others, remains an open question.
cRBM allowed us to compute the effective, functional connections between each pair of neurons, aggregated to functional connections between each pair of regions, by perturbing neural activity in silico. Importantly, we found that this regionscale connectivity is wellconserved across specimen. This observation is nontrivial because each recording only lasted ∼25 min, which represents a short trajectory across accessible brain states. It suggests that, although each individual brain may be unique at the neuronal scale, the functional organization could be highly stereotyped at a sufficiently coarsegrained level.
It would be naive to assume that these functional connections equate biophysical, structural connections (Das and Fiete, 2020). Both represent different, yet interdependent aspects of the brain organization. Indeed, we found that structural connectivity is wellcorrelated to functional connectivity, confirming that functional links are tied to the structural blueprint of brain connectivity (Figure 6). Furthermore, strong (weak) functional connections are predictive of present (absent) structural connections between brain regions, although intermediate values are ambiguous.
It will be crucial to synergistically merge structural and dynamic information of the brain to truly comprehend brainwide functioning (Bargmann and Marder, 2013; Kopell et al., 2014). Small brain organisms are becoming an essential means to this end, providing access to a relatively large fraction of cells (Ahrens and Engert, 2015). To generate new scientific insights it is thus essential to develop analytical methods that can scale with the rapidly growing size of both structural and dynamic data (Helmstaedter, 2015; Ahrens, 2019). In this study, we have established that the cRBM can model highdimensional data accurately, and that its application to zebrafish recordings was crucial to unveil their brainscale assembly organization. In future studies, cRBMs could be used to generate artificial data whose statistics replicate those of the zebrafish brain. This could be used for further in silico ablation and perturbation studies with strong physiological footing, crucial for developing hypotheses for future experimental work (Jazayeri and Afraz, 2017; Das and Fiete, 2020). Lastly, the application of cRBMs is not specific to calcium imaging data, and can therefore be readily applied to highdimensional neural data obtained by other recording techniques.
Materials and methods
Data and code availability
Request a detailed protocolThe cRBM model has been developed in Python 3.7 and is available at: https://github.com/jertubiana/PGM, (copy archived at swh:1:rev:caf1d9fc545120f7f1bc1420135f980d5fd6c1fe; Tubiana and van der Plas, 2023). An extensive example notebook that implements this model is provided here.
Calcium imaging data preprocessing was performed in MATLAB (Mathworks) using previously published protocols and software (Panier et al., 2013; Wolf et al., 2017; Migault et al., 2018; Tubiana et al., 2020). The functional data recordings, the trained cRBM models and the structural and functional connectivity matrix are available at https://gin.gnode.org/vdplasthijs/cRBM_zebrafish_spontaneous_data.
Figures of neural assemblies or neurons (Figures 1 and 3) were made using the Fishualizer, which is a 4D (space +time) data visualization software package that we have previously published (Migault et al., 2018), available at https://bitbucket.org/benglitz/fishualizer_public. Minor updates were implemented to tailor the Fishualizer for viewing assemblies, which can be found at https://bitbucket.org/benglitz/fishualizer_public/src/assembly_viewer.
All other data analysis and visualization was performed in Python 3.7 using standard packages (numpy Harris et al., 2020), scipy (Virtanen et al., 2020), scikitlearn (Pedregosa, 2011), matplotlib (Hunter, 2007), pandas (McKinney, 2010), seaborn (Waskom, 2021), h5py. The corresponding code is available at https://github.com/vdplasthijs/zfrbm (copy archived at swh:1:rev:b5df4e37434c0b18120485b8d856596db0b92444; van der Plas, 2023).
Zebrafish larvae
Request a detailed protocolExperiments were conducted on nacre mutants, aged 5–7 days postfertilization (dpf). Larvae were reared in Petri dishes at 28 °C in embryo medium (E3) on a 14/10 hr light/dark cycle, and were fed powdered nursery food every day from 6 dpf. They were expressing either the calcium reporter GCaMP6s (fish 1–4, 6, and 8) or GCaMP6f (fish 5 and 7) under the control of the nearly panneuronal promoter elavl3 expressed in the nucleus Tg(elavl3:H2BGCaMP6). Both lines were provided by Misha Ahrens and published by Vladimirov et al., 2014 (H2BGCaMP6s) and Quirin et al., 2016 (H2BGCaMP6f). Experiments were approved by Le Comité d’Éthique pour l’Experimentation Animale Charles Darwin C2EA05 (02601.01).
Lightsheet microscopy of zebrafish larvae
Request a detailed protocolSpontaneous neural activity (i.e. in the absence of sensory stimulation) was recorded in larval zebrafish using lightsheet microscopy, which acquires brainscale scans by imaging multiple $z$planes sequentially (Panier et al., 2013; Wolf et al., 2017; Migault et al., 2018). Larvae were placed in 2% low melting point agarose (SigmaAldrich), drawn tailfirst into a glass capillary tube with 1 mm inner diameter via a piston and placed in chamber filled with E3 in the microscope. Recordings were of length 1514 ± 238 seconds (mean ± standard deviation), with a brain volume imaging frequency of 3.9 ± 0.8 Hz.
The following imaging preprocessing steps were performed offline using MATLAB, in line with previously reported protocols (Panier et al., 2013; Migault et al., 2018). Automated cell segmentation was performed using a watershed algorithm (Panier et al., 2013; Migault et al., 2018) and fluorescence values of pixels belonging to the same neuron was averaged to obtain cell measurements. The fluorescence intensity values $F$ were normalized to $\mathrm{\Delta}F/F=(F\u27e8F\u27e9)/(\u27e8F\u27e9{F}_{0})$ where $\u27e8F\u27e9$ is the baseline signal per neuron and F_{0} is the overall background intensity (Migault et al., 2018). The $\mathrm{\Delta}F/F$ activity of different imaging planes was subsequently temporally aligned using interpolation (because of the time delay between imaging planes; Migault et al., 2018) and deconvolved to binarized spike traces using Blind Sparse Deconvolution (BSD) (Tubiana et al., 2020). BSD estimates the most likely binary spike trace by minimizing the L_{2} norm of the difference between the estimated spike trace convolved with an exponential kernel and the groundtruth calcium data, using L_{1} sparsity regularization and online hyperparameter optimization. Calcium kernel time constants used for deconvolution were inferred using BSD on the spontaneous activity of three different fish (approximately 5000 neurons per fish, recorded at 10 Hz, previously reported by Migault et al., 2018). For the GCaMP6s line, we used a rise time of 0.2 s and a decay time of 3.55 s; for the GCaMP6f line, we used 0.15 s and 1.6 s, respectively.
Brain activity was recorded of 15 animals in total. Of these recordings, 1 was discarded because of poor image quality and 6 were discarded because neurons were inactive (defined by less than 0.02 spikes/(neurons × time points)), hence leaving 8 data sets for further analysis. The recorded brains were then registered onto the ZBrain Atlas (Randlett et al., 2015) and the mapzebrain atlas (Kunst et al., 2019) for anatomical labeling of neurons (Migault et al., 2018). The ZBrain Atlas was used in Figures 1—4 because of its detailed region descriptions (outlining 294 regions in total). However, we also registered our data to the mapzebrain atlas (72 regions in total) in order to compare our results with the structural connectivity matrix which was defined for this atlas only (Kunst et al., 2019). Only neurons that were registered to at least 1 ZBrain region were used for analysis (to filter imaging artefacts). This resulted in $40709\pm 13854$ neurons per recording (mean ± standard deviation, minimum = 23446, maximum = 65517).
Maximum entropy principle
Here, we provide in brief the general derivation of the class of maximum entropy probabilistic models. Restricted Boltzmann Machines are an instance of this model, which is detailed in the following sections. The maximum entropy principle is used to create probabilistic models $P(x)$ (where $x$ denotes one data configuration sample) that replicate particular data statistics f_{k}, but are otherwise most random, and therefore least assumptive, by maximizing their entropy $H={\sum}_{x}P(x)\mathrm{log}(P(x))$(Gardella et al., 2019). The goal of the model is to match its model statistics ${\u27e8{f}_{k}\u27e9}_{\text{model}}={\sum}_{x}P(x){f}_{k}(x)$ to the empirical data statistics ${\u27e8{f}_{k}\u27e9}_{\text{data}}={F}_{k}$. This is done using Lagrange multipliers ${\mathrm{\Lambda}}_{k}$:
which yields, when $\stackrel{~}{H}$ is maximized with respect to $P(x)$, the Boltzmann distribution (see, e.g., Bialek, 2012 for a full derivation):
where $E(x)$ is defined as the resulting energy function. Importantly, the data dependency (${F}_{k}$) disappears when going from Equation 5 to Equation 6. Hence, the maximum entropy principle only defines the shape of the distribution $P(x)$, but not its specific parameters ${\mathrm{\Lambda}}_{k}$ (Bialek, 2012). In the case of RBM, these are then optimized using maximum likelihood estimation, as detailed in the sections below.
Motivation for choice of statistics
Request a detailed protocolThe derivation above describes the general maximum entropy model for a set of statistics $\{{f}_{k}\}$. The objective of this study is to extract the assembly structure from neural data, therefore creating two layers: a visible (neural data) layer $\text{v}=({v}_{1},{v}_{2},\mathrm{\dots},{v}_{N})$ and a hidden (latent) layer $\text{h}=({h}_{1},{h}_{2},\mathrm{\dots},{h}_{M})$. The model should capture the mean activity of each neuron $\u27e8{v}_{i}\u27e9$, their pairwise correlations $\u27e8{v}_{i}{v}_{j}\u27e9$, the neuronHU interactions $\u27e8{v}_{i}{h}_{\mu}\u27e9$ and a function of ${h}_{\mu}$. The latter is determined by the potential $\mathcal{U}$, which we set to be a double Rectified Linear Unit (dReLU), as motivated in the following sections. Fitting all ${N}^{2}$ pairwise interactions $\u27e8{v}_{i}{v}_{j}\u27e9$ is computationally infeasible, but under the cell assembly hypothesis we assume that this should not be necessary because collective neural behavior is expected to be explained by membership to similar assemblies via $\u27e8{v}_{i}{h}_{\mu}\u27e9$, and can therefore be excluded. We later show that pairwise correlations are indeed optimized implicitly (Figure 2). All other statistics are included and therefore explicitly optimized, also see Equation 14.
Restricted Boltzmann machines
A Restricted Boltzmann Machine (RBM) is an undirected graphical model defined on a bipartite graph (Smolensky, 1986; Hinton, 2002; Hinton and Salakhutdinov, 2006), see Figure 2A. RBMs are constituted by two layers of random variables, neurons $\mathbf{\text{v}}$ and Hidden Units (HUs) $\mathbf{\text{h}}$, which are coupled by a weight matrix $\mathbf{W}$. There are no direct couplings between pairs of units within the same layer. Here, each visible unit v_{i} corresponds to a single recorded neuron with binary (spikedeconvolved) activity (${v}_{i}(t)\in \{0,1\}$). Each Hidden Unit (HU) ${h}_{\mu}$ corresponds to the (weighted) activity of its neural assembly and is chosen to be realvalued. The joint probability distribution $P(\text{v},\text{h})$ writes (Hinton and Salakhutdinov, 2006; Tubiana and Monasson, 2017):
where $E$ is the energy function and $Z={\sum}_{\mathbf{v}}{\int}_{\mathbf{h}}dvdh\cdot \mathrm{exp}\left(E(\text{v},\text{h})\right)$ is the partition function. The weights g_{i} and potentials $\mathcal{U}}_{\mu$ control the activity level of the visible units and the marginal distributions of the HUs respectively, and the weights ${w}_{i,\mu}$ couple the two layers. Note that while $\mathbf{\text{v}}$ is directly observed from the neural recordings, $\mathbf{\text{h}}$ is by definition unobserved (i.e. hidden) and is sampled from the observed $\mathbf{\text{v}}$ values instead.
From data to features
Request a detailed protocolGiven a visible layer configuration $\mathbf{\text{v}}$, a HU ${h}_{\mu}$ receives the input ${I}_{\mu}(\mathbf{v})={\sum}_{i}{w}_{i\mu}{v}_{i}\equiv \mathbf{w}_{\mu}{}^{T}\mathbf{v}$ and, owing to the bipartite architecture, the conditional distribution $P(\text{h}\text{v})$ factorizes as:
where ${\mathrm{\Gamma}}_{\mu}(I)=\mathrm{log}\left({\int}_{h}\phantom{\rule{thinmathspace}{0ex}}dh\cdot \mathrm{exp}\left({\mathcal{U}}_{\mu}(h)+hI\right)\right)$ is the cumulant generating function associated to the potential $\mathcal{U}}_{\mu$ that normalizes Equation 8 (Tubiana et al., 2019b). The average activity of HU ${h}_{\mu}$ associated to a visible configuration $\mathbf{\text{v}}$ is given by a linearnonlinear transformation (as defined by the properties of the cumulant generating function):
Throughout the manuscript, we use this definition to compute HU activity ${h}_{\mu}(t)=\u27e8{h}_{\mu}\mathbf{v}(t)\u27e9$ (e.g., in Figure 4).
From features to data
Request a detailed protocolConversely, given a hidden layer configuration $\mathbf{\text{h}}$, a visible unit v_{i} receives the input ${I}_{i}(\mathbf{h})={\sum}_{\mu}{w}_{i,\mu}{h}_{\mu}\equiv \mathbf{w}_{\mathbf{i}}{}^{T}\mathbf{h}$ and the conditional distribution factorizes as:
and the average sampled v_{i} activity is given by:
where $\sigma (x)=1/(1+{e}^{x})$ is the logistic function. Hence, a sampled visible layer configuration $\mathbf{\text{v}}$ is obtained by a weighted combination of the HU activity followed by Bernoulli sampling. RBMs are generative models, in the sense that they can generate new, artificial data using Equations 8 and 10. Figure 2B illustrates this Markov Chain Monte Carlo (MCMC) process, by recursively sampling from $P(\mathbf{h}\mathbf{v})$ and $P(\mathbf{v}\mathbf{h})$, which converges at equilibrium to $P(\text{v},\text{h})$.
Marginal distributions
Request a detailed protocolThe marginal distribution $P(\mathbf{v})$ has a closedform expression because of the factorized conditional distribution of Equation 9 (Tubiana et al., 2019a; Tubiana et al., 2019b):
For a quadratic potential $\mathcal{U}}_{\mu}(h)=\frac{{\gamma}_{\mu}{h}_{\mu}^{2}}{2}+{\theta}_{\mu}{h}_{\mu$, the cumulant generating function would also be quadratic and $P(\mathbf{v})$ would reduce to a Hopfield model, that is, a pairwise model with an interaction matrix ${J}_{ij}={\sum}_{\mu}\frac{{w}_{i\mu}{w}_{j\mu}}{{\gamma}_{\mu}}$ (Tubiana et al., 2019a). Otherwise, ${\mathrm{\Gamma}}_{\mu}$ is not quadratic, yielding highorder effective interaction terms between visible units and allowing RBMs to express more complex distributions. Importantly, the number of parameters remains limited, controlled by $M$ and does not scale as ${N}^{2}$ (unlike pairwise models).
Choice of HU potential
Request a detailed protocolThe choice of HU potential determines three related properties: the HU conditional distribution $P(\mathbf{h}\mathbf{v})$, the transfer function of the HUs and the parametric form of the marginal distribution $P(\mathbf{v})$. Hereafter we use the doubleRectified Linear Unit (dReLU) potential:
Varying the parameters $\{{\gamma}_{\mu ,+},{\gamma}_{\mu ,},{\theta}_{\mu ,+},{\theta}_{\mu ,}\}$ allows the potential to take a variety of shapes, including quadratic potentials (${\gamma}_{\mu ,+}={\gamma}_{\mu ,}$, ${\theta}_{\mu ,+}={\theta}_{\mu ,}$), ReLU potentials $({\gamma}_{\mu ,}\to \mathrm{\infty})$ and doublewell potentials (Tubiana et al., 2019b). The associated cumulant generating function $\mathrm{\Gamma}(I)$ is nonquadratic in general, and depending on the parameters, the transfer function can be linear, ReLUlike (asymmetric slope and thresholding) or logisticlike (strong local slopes for binarizing inputs). Closedform expressions of $\mathrm{\Gamma}$ are detailed in Tubiana et al., 2019a; Tubiana et al., 2019b, and its derivatives are also detailed in Tubiana, 2018, p4950. Note that the dReLU potential $\mathcal{U}}_{\mu$ and distribution $P(\mathbf{v})$ are invariant to the sign swap transformation ${\gamma}_{\mu ,+},{\theta}_{\mu ,+}\u27fa{\gamma}_{\mu ,},{\theta}_{\mu ,}$ and ${w}_{i\mu}\u27fa{w}_{i\mu}\forall i,\mu $ (leading to ${h}_{\mu}\u27fa{h}_{\mu}$). For visual clarity, we perform this sign swap transformation after training on all HUs with predominantly negative weights (defined by $\sum _{i}{w}_{i,\mu}<0$). Subsequently all HUs are positively activated if the group of neurons to which it connects is strongly active.
RBM training
Request a detailed protocolThe RBM is trained by maximizing the average loglikelihood of the empirical data configurations $\mathcal{L}={\u27e8\mathrm{log}P(\mathbf{v})\u27e9}_{\text{data}}$, using stochastic gradient descent methods. The gradient update steps are derived by calculating the derivative of $\mathcal{L}$, using Equation 12, with respect to the model parameters (Tubiana et al., 2019a):
Each gradient of $\mathcal{L}$ is thus the difference between a data statistic ${\u27e8{f}_{k}\u27e9}_{\text{data}}$ and a model statistic ${\u27e8{f}_{k}\u27e9}_{\text{model}}$. Hence the model learns to match these statistics to the training data. Importantly, model statistics ${\u27e8{f}_{k}\u27e9}_{\text{model}}$ cannot be evaluated exactly due to the exponentially large number of data configurations (e.g. ${2}^{N}$ visible configurations). Therefore they are approximated by computing the statistics of modelgenerated data using the MCMC sampling scheme defined with Equations 8 and 10 (see Materials and methods  ‘Matching data statistics to model statistics’ for more detail). MCMC sampling of a Boltzmann distribution in such highdimensional space is in general very challenging owing to the exponentially long time to reach equilibrium. We use the persistent contrastive divergence approximation (Tieleman, 2008) and discuss its validity below.
Compositional restricted boltzmann machine
Request a detailed protocolIn the previous sections, we have described the general properties of RBMs. We now motivate the specific RBM model choices that we have implemented, such as the dReLU potential and sparsity regularization, by discussing their impact on the properties of RBMgenerated data.
Directed graphical models, for example, PCA, ICA, sparse dictionaries or variational autoencoders, prescribe a priori statistical constraints for their data representations, such as orthogonality/independence or specific marginal distributions such as Gaussian/sparse distributions. In contrast, the statistical properties of the representation of the data learned by RBMs are unknown a priori by construction (because of the maximum entropy principle). Instead, they emerge from the structure of the weight matrix, the potentials and the recursive backandforth sampling procedure described above. We have therefore previously studied the properties of typical samples of RBM with random weights as a function of the visible and hidden unit potentials and properties of the weight matrix using statistical mechanics tools (Tubiana and Monasson, 2017; Tubiana et al., 2019a). We have identified the three following typical behaviors, or phases.
In the ferromagnetic phase, a typical sample from $P(\text{v},\text{h})$ has a single strongly activated HU ($m(t)\sim 1$, where $m(t)$ is the number of activated HUs at time $t$), whereas the others are not or merely weakly activated. The corresponding active visible units v_{i} are defined by the weight vector ${\text{w}}_{{\mu}^{\star}}$ associated to the active HU ${h}_{{\mu}^{\star}}$ (see Equation 10).
In the spinglass phase, a typical sample does not have any relatively strongly activated HUs, but instead many moderately activated ones ($m(t)\sim M$). They interfere in a complex fashion to produce different visible unit configurations and there is no clear correspondence between the weight matrix and a typical data configuration.
Finally, in the compositional phase, a typical sample from $P(\text{v},\text{h})$ has a small number of strongly activated HUs ($1\ll m(t)\ll M$) whereas the others are weak or silent. Their weights are linearly combined through Equation 10 to produce the corresponding visible layer configuration. The compositional phase is desirable because, firstly, there exists a simple link between the weight matrix and typical data configurations (they are obtained by combining a few weights), which facilitates interpretation of biological systems (Tubiana et al., 2019b). Secondly, the corresponding neural activity distribution is rich, as different choices of HU subsets yield a combinatorial diversity of visible layer configurations. Moreover, the modular nature of the compositional phase facilitates the assembly organization of neural dynamics, as motivated in the Introduction.
A set of sufficient conditions for the emergence of the compositional phase are (Tubiana and Monasson, 2017):
The HUs are unbounded and realvalued with a nonlinear, ReLUlike transfer function.
The weight matrix $\mathbf{\text{W}}$ is sparse.
The columns ${\mathbf{w}}_{\mu}$ of the weight matrix have similar norm. (If a weight column associated to one HU is much larger than the others, visible configurations are solely aligned to it according to Equation 10.)
The first condition is satisfied by the dReLU potential (but not by quadratic potentials or binaryvalued HUs). The second condition is enforced in practice by adding a L_{1} sparse penalty term $\lambda \cdot {\sum}_{i\mu}{w}_{i\mu}$ to the loglikelihood cost function. In our experiments, the optimal sparsity parameter $\lambda $ was determined to be $\lambda =0.02$ by crossvalidation (Figure 2—figure supplement 1). The final condition is achieved by enforcing that $\mathrm{Var}({h}_{\mu})=1$ and $\u27e8{h}_{\mu}\u27e9\sim 0\forall \mu $. This is done by an appropriate reparameterization of the HU potential of Equation 13 and a batchnorm–like procedure, described in detail in Tubiana, 2018. This normalization promotes homogeneity among HU importance, preventing some units from being disconnected or others from dominating. In addition, ensuring that ${h}_{\mu}=\mathcal{O}(1)$ irrespective of the visible layer size (as opposed to e.g., $\frac{1}{2}({\gamma}_{+}+{\gamma}_{})=1$ which yields ${h}_{\mu}\sim \sqrt{N}$) avoids the problem of illconditioned Hessians that was previously described by Hinton, 2012.
To emphasise the departure from the classic RBM formulation in this study, we name our model compositional RBM (cRBM).
Algorithmic implementation
In the previous sections, we have described the cRBM model in full mathematical detail. The corresponding algorithmic implementation was adapted from Tubiana et al., 2019b. In addition, we have made several major implementation and algorithmic changes to accommodate the large data size of the zebrafish neural recordings. We provide the code opensource, and describe the code improvements and hyperparameter settings in this section. The following improvements were made, leading to a substantial reduction of computation time:
Python 3 and numba (Lam et al., 2015) were used to compile custom functions, enabling SIMD vectorization and multicore parallelism.
The sampling of $P({h}_{\mu}{I}_{\mu})$ and evaluating its cumulant generating function ${\mathrm{\Gamma}}_{\mu}$ and various moments requires repeated and costly evaluation of error functions erf and related functions (Tubiana, 2018, p4950). Fast numerical approximations of these functions were implemented based on Abramowitz et al., 1988 (p299).
The number of memory allocation operations was minimized.
The optimization algorithm was changed from stochastic gradient ascent to RMSprop (i.e. ADAM without momentum) with learning rate $5\cdot {10}^{4}$ to $5\cdot {10}^{3}$, ${\beta}_{1}=0$, ${\beta}_{2}=0.999$, $\u03f5={10}^{6}$, see Kingma and Ba, 2014 for a definition of the parameters. Compared to the original stochastic gradient ascent, the adaptive learning rates of RMSprop/ADAM yield larger updates for the weights attached to neurons with very sparse activity, resulting in substantially faster convergence.
Hyperparameter settings
Request a detailed protocolThe following hyperparameters were used in the experiments of this manuscript:
Number of hidden unit $M$: 200. This value was determined by crossvalidation (Figure 2—figure supplement 1) on one data set (example fish #3). Because this crossvalidation procedure was computationally expensive, the same value was used for all other data sets, except for 3 data sets which used $M=100$ because their $N\approx \frac{1}{2}{N}_{\mathrm{\#}3}$.
Sparse regularization penalty $\lambda $: 0.02 (determined by crossvalidation).
Batch size: 100, 200, or 400. Larger batch sizes yield longer training time but more stable training; batch size was increased if training failed to converge.
Number of Monte Carlo chains: 100.
Number of gradient updates: $2\cdot {10}^{5}$.
Number of Monte Carlo steps between each gradient update: 15.
Initial learning rate $\eta $: between $5\cdot {10}^{4}$ and $5\cdot {10}^{3}$. We used $5\cdot {10}^{3}$ by default and if weight divergence was observed, the learning was reinitialized with a reduced learning rate. This occurred notably for high$M$ and low$\lambda $ models during the crossvalidation procedure of Figure 2—figure supplement 1.
Learning rate annealing scheme: the learning rate geometrically decayed during training, starting after 25% of the gradient update steps, from its initial value $\eta $ to a final value of ${10}^{5}$.
Number of training data samples: 70% of frames of each recording (=4086 training data samples on average), see section ‘Train / test data split’ for details.
Computational limitations
Request a detailed protocolWe found that 57.5% ($=23/40$) of cRBMs with optimal $(\lambda ,M)$ settings successfully converged. cRBM models of these zebrafish data sets could be estimated in approximately 8–12 hr using 16 CPU threads (Intel Xeon Phi processor). The $(\lambda ,M)$crossvalidation was therefore completed in three weeks using two desktop computers. Previously, we observed that this model requires a fixed number of gradient updates to converge, rather than a fixed number of epochs (Tubiana et al., 2019a; Tubiana et al., 2019b; Bravi et al., 2021). Hence, in principle, runtime does not strictly depend on the recording length, as the number of epochs can be reduced for longer recordings (assuming that the data distribution remains statistically stationary).
Validity of the persistent contrastive divergence algorithm
Request a detailed protocolTraining RBM requires extensive MCMC sampling which is notoriously difficult for highdimensional data sets. We resolve this by using Persistent Contrastive Divergence (PCD) to approximate the gradients (Tieleman, 2008). In this section, we discuss why this worked to successfully converge, despite the very large data size.
The typical number of Monte Carlo steps required to transition from one energy minimum to another through an energy barrier $\mathrm{\Delta}E$ follows the Arrhenius law, scaling as ${e}^{\mathrm{\Delta}E}$. In the thermodynamic limit ($N\to \mathrm{\infty}$), $\mathrm{\Delta}E$ scales as the system size $N$ multiplied by the typical energy required to flip a single visible unit, corresponding here to the inputs received from the hidden layer $I$. In contrast, for PCD only a limited number of MC steps (here, 15) are applied between each gradient update. Three factors explain why reasonably successful convergence was achieved in the trainings presented here.
Firstly, the use of the L_{1} regularization limits the magnitude of the weights and therefore limits the input scale $I$. Secondly, in the compositional phase, the energy barriers do not scale as the full system size $N$ but rather as the size of one assembly $pN$ where $p$ is the fraction of nonzero weights (Tubiana and Monasson, 2017). Indeed, transitioning from one energy minimum, characterized by a subset of strongly activated HUs, to another minimum, characterized by another set of strongly activated HUs, is done by gradually morphing the first set into the second (Roussel et al., 2021). Compared to a direct transition, such a path is favored because the intermediate states are thermodynamically stable and energy barriers are smaller as each HU flip has an energy cost $\sim pN$. Lastly, throughout PCD training, MCMC sampling is not performed at thermal equilibrium and the model updates of the parameters of the distribution promote mixing (Tieleman and Hinton, 2009). This is seen from Equation 14: the loglikelihood gradient is the difference between the gradient of the energy averaged over the empirical data and the energy averaged over MCMC samples. Ascending the gradient amounts to pushing down the energy of data configurations and pushing up the energy of MCMC samples, thereby promoting mixing of the Markov chains.
Overall, combining small learning rates (and large number of gradient updates), large regularization, large number of Markov Chains and Monte Carlo steps has allowed convergence to be reached for the majority of cRBM training sessions.
Functional connectivity inference
Effective connectivity matrix
Request a detailed protocolIn this section, we present a derivation of the effective coupling matrix between neurons from the marginal distribution $P(\mathbf{v})$ using cRBMs. This is achieved by perturbing the activity of each neuron individually and quantifying the effect on other neurons. We first define the local coupling ${J}_{ij}$ between two neurons v_{i} and v_{j} for a generic probability distribution $P({v}_{1},{v}_{2},\cdots ,{v}_{N})$, given a data configuration $\mathbf{v}$:
In other words, ${J}_{ij}$ is defined as the impact of the state of neuron $j$ on neuron $i$ in the context of activity pattern $\mathbf{v}$. Hence, the effective connectivity matrix $\mathbf{\text{J}}$ mathematically defines the functional connections, which can only be done using a probabilistic model $P(\text{v})$. A positive (negative) coupling ${J}_{ij}$ indicates correlated (anticorrelated) collective behavior of neurons $i,j$. This effective coupling value is symmetric (because of Bayes’ rule): ${J}_{ij}(\mathbf{v})={J}_{ji}(\mathbf{v})$. For context, note that ${J}_{ij}(\mathbf{v})$ is uniformly zero for an independent model of the form $P({v}_{1},\cdots ,{v}_{N})=\prod _{i}{P}_{i}({v}_{i})$, and that for a maximum entropy pairwise (Ising) model, with $P({v}_{1},\cdots ,{v}_{N})=\frac{1}{Z}\mathrm{exp}\left(\sum _{i}{g}_{i}{v}_{i}+\sum _{i<j}{J}_{ij}^{\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}}{v}_{i}{v}_{j}\right)$, the ${J}_{ij}(\mathbf{v})$ matrix exactly identifies with the coupling matrix ${J}_{ij}^{\mathrm{ising}}$, and does not depend on the data configuration $\mathbf{\text{v}}$ (so ${J}_{ij}(\mathbf{v})={J}_{ij}$).
However, in general, and for RBMs in particular, ${J}_{ij}(\mathbf{v})$ depends on the data set $\mathbf{\text{v}}$, and an overall coupling matrix can be derived by taking its average over all data configurations:
Although Equation 16 has a closedform solution for RBMs (by inserting Equation 12), a naive evaluation requires $\mathcal{O}({N}^{3}MT)$ operations where $T$ is the number of data samples. However, a fast and intuitive approximation can be derived by performing a secondorder Taylor expansion of ${\mathrm{\Gamma}}_{\mu}({I}_{\mu})$:
Equation 17 is exact for quadratic potential and in general justified as the contribution of neurons $i,j$ is small compared to the scale of variation of ${\mathrm{\Gamma}}_{\mu}$, $\mathcal{O}(\sqrt{pN})$ where $p$ is the fraction of nonzero couplings. In conclusion, we have mathematically derived the effective coupling between any two neurons $i$ and $j$. Intuitively, two neurons $i,j$ are effectively connected if they are connected to the same HUs (Equation 17).
From interneuron to interregion connectivity
Request a detailed protocolIn the above section, we have derived the interneuronal connectivity matrix $\mathbf{\text{J}}$. This matrix is then aggregated to an interregional connectivity matrix ${\text{J}}^{\mathcal{R}}$ by taking the normalised L_{1}norm of the corresponding $\mathbf{\text{J}}$ matrix block elements (i.e., ${J}_{km}^{\mathcal{R}}={\sum}_{i\in {R}_{k},j\in {R}_{m}}{J}_{ij}/({N}_{{R}_{k}}\cdot {N}_{{R}_{m}})$, where ${R}_{k}$ is the set of neurons in region $k$).
Next, to derive the average connectivity matrix across multiple recordings, we used a weighted average of the individual recordings, with a regionpair specific weight equal to the length of the recording multiplied by the sum of the number of neurons in both regions (also see Section  ‘Specimen averaging of connectivity matrices’). Compared to a naive average, this weighted average accounts for the variable number of neurons per region between recordings.
Training cRBM models for connectivity estimates
Request a detailed protocolConstructing the functional connectivity matrix of a cRBM does not require test data, but just the estimated weight matrix $\mathbf{\text{W}}$ (as explained above). Therefore we trained new cRBMs using the entire recordings (100% of data) to fully use the information available. cRBM training is stochastic, and to mitigate the possible variability that could arise we trained five cRBMs for each recording. Then, to assess convergence, we selected all cRBMs with $0.01<\text{std}(\mathbf{\text{w}})<0.1$, where std denotes standard deviation, for further functional connectivity analysis (yielding 23 cRBMs for 8 data sets in total). Connectivity estimates of multiple cRBM models per data sets were averaged.
Connectivity inference baselines
Request a detailed protocolWe considered four additional connectivity inference baseline methods:
The covariance matrix.
The Pearson correlation matrix.
The sparse inverse covariance matrix inferred by graphical LASSO (Friedman et al., 2008) (as implemented in scikitlearn with default settings Pedregosa, 2011). Graphical LASSO is an efficient method for inference of large scale connectivity. Unfortunately, the implementation available failed to converge in reasonable time due to the high dimensionality of the data.
The Ising model with pseudolikelihood maximization (PLM) inference (Ravikumar et al., 2010).
Results obtained with the covariance and correlation matrices are presented in Figure 6—figure supplement 1. The connectivity matrices obtained by the PLM Ising model (not shown) correctly identified the diagonal entries of the regionregion matrix, but not the offdiagonal coefficients and had a weaker correlation with the structural connectivity matrix than the covariance and correlation matrices (${r}_{S}=0.06$ using 4 fish).
Optimizing the free parameters of cRBM
We set the free parameters $\lambda $ (sparsity regularization parameter) and $M$ (number of HUs) by crossvalidating a large range of $(\lambda ,M)$ values for one data set (fish #3). This was done by training cRBMs on 70% of the data, and evaluating model performance on the remaining test data, as detailed below. The resulting optimal values could then be used for all data sets (where $M$ was scaled with the number of neurons $N$). Importantly, the $(\lambda ,M)$ parameters implicitly tune the average assembly size. Increasing the number of HUs and/or increasing the regularization strength decreases the average number of neurons per assembly (Tubiana et al., 2019a). Intuitively, assemblies that are too small do not have the capacity to capture highorder correlations, while assemblies that are too large would fail to account for local coactivations. Hence, the $(M,\lambda )$crossvalidation effectively identifies the optimal assembly sizes that fit the data statistics.
Train / test data split
Request a detailed protocolWe split up one recording (fish #3) into training data (70% of recording) and withheld test data (30% of recording) for the free parameter ($\lambda ,M$) optimization procedure. This enabled us to assess whether the cRBMs learned to model the data statistics (as described in the main text, Figure 2, Figure 2—figure supplement 1), while ensuring that the cRBMs are not overfitted to the specific training data configurations. Importantly, this assumes that the test data comes from the same statistical distribution as the training data (while consisting of different data configurations). To ensure this, we split up the recording of example fish #3 (used for parameter optimization) in training and test splits as follows (before training the cRBMs): We divided the recording of length $T$ in 10 chronological segments of equal length (so that segment 1 has time points $\{t\in [1,\frac{T}{10})\}$ et cetera), with the rationale that by maintaining temporal order within each segment we would later be able to conduct dynamic activity analysis. This yielded $\left(\genfrac{}{}{0pt}{}{10}{3}\right)=120$ possible training/test splits of the neural data. We then evaluated the statistical similarity between the training and test split of each combination, by assessing the mean neural activity $\u27e8{v}_{i}\u27e9$ and pairwise neural correlations $\u27e8{v}_{i}{v}_{j}\u27e9\u27e8{v}_{i}\u27e9\u27e8{v}_{j}\u27e9$ statistics. We quantified the similarity between training and test statistics by calculating the Root Mean Square Error ($\text{RMSE}({\text{x}}_{1},{\text{x}}_{2})=\sqrt{\frac{1}{{N}_{x}}{\sum}_{n=1}^{{N}_{x}}{\left({x}_{1}(n){x}_{2}(n)\right)}^{2}}$). The most similar split is defined by the lowest RMSE, but to show that cRBM are not dependent on picking the best possible split, but rather on avoiding the bad splits, we then chose to use the split with the ${10}^{\text{th}}$percentile ranking RMSE. We hope that this aids future studies, where a potentially high number of possible splits prevents researchers from evaluating all possible splits, but a good split may nevertheless be found efficiently.
Assessment of data statistics
Request a detailed protocolPlease note that the loss function, the loglikelihood, is computationally intractable and therefore cannot be readily used to monitor convergence or goodnessoffit after training (Fischer & Igel and Igel, 2012). Moreover, approximations of the loglikelihood based on annealed importance sampling were found to be unreliable due to the large system size. However, because (c)RBM learn to match data statistics to model statistics (see Materials and methods–RBM training), we can directly compare these to assess model performance. Therefore, we assessed the following quantities.
Matching data statistics to model statistics
Request a detailed protocolFirstly, we evaluated three statistics that cRBMs are trained to optimize: the mean activity of neurons $\u27e8{v}_{i}\u27e9$, the mean activity of HUs $\u27e8{h}_{\mu}\u27e9$ and their pairwise interactions $\u27e8{v}_{i}{h}_{\mu}\u27e9$. Additionally, second order statistics of pairwise neuronneuron interactions $\u27e8{v}_{i}{v}_{j}\u27e9$, HUHU interactions $\u27e8{h}_{\mu}{h}_{\nu}\u27e9$ and the reconstruction quality were evaluated, which the cRBM was not constrained to fit. Monitoring HU single and pairwise statistics $\u27e8{h}_{\mu}\u27e9$ and $\u27e8{h}_{\mu}{h}_{\nu}\u27e9$ served two purposes: (i) validation of model convergence and (ii) assessing whether correlations between assemblies can be captured by this bipartite model (i.e., without direct couplings between hidden units or an additional hidden layer). For each statistic $\u27e8{f}_{k}\u27e9$, we computed its value based on empirical data ${\u27e8{f}_{k}\u27e9}_{\text{data}}$ and on the model ${\u27e8{f}_{k}\u27e9}_{\text{model}}$, which we then quantitatively compared to assess model performance.
Data statistics ${\u27e8{f}_{k}\u27e9}_{\text{data}}$ were calculated on withheld test data (30% of recording). Naturally, the neural recordings consisted only of neural data $\mathbf{\text{v}}$ and not of HU data $\mathbf{\text{h}}$. We therefore computed the expected value of ${\text{h}}_{t}$ at each time point $t$ conditioned on the empirical data ${\text{v}}_{t}$, as further detailed in Methods  ‘From data to features’.
Model statistics ${\u27e8{f}_{k}\u27e9}_{\text{data}}$ cannot be calculated exactly, because that would require one to sample all possible states $P(\text{v},\text{h})$, and were therefore approximated by evaluating cRBMgenerated data. Here, 300 Monte Carlo chains were each initiated on random training data configurations and 50 configurations were sampled consecutively for each chain, with 20 sampling steps between saved configurations, after a burnin period of 100 effective sampling configurations.
The $\u27e8{v}_{i}{h}_{\mu}\u27e9$ statistic (Figure 2C) was corrected for the sparsity regularization, by adding the sparsity regularization parameter $\lambda$ to $\u27e8{v}_{i}{h}_{\mu}\u27e9:\u27e8{v}_{i}{h}_{\mu}{\u27e9}_{\text{model}}=\u27e8{v}_{i}{h}_{\mu}{\u27e9}_{\text{modelgenerated data}}+\lambda \cdot sign({w}_{i,\mu})$. Furthermore, $({v}_{i},{h}_{\mu})$ pairs with exactly ${w}_{i,\mu}=0$ were excluded from analysis (5% of total for optimal cRBM in Figure 2C).
The pairwise neuronneuron and HUHU statistics ($\u27e8{v}_{i}{v}_{j}\u27e9$, $\u27e8{h}_{\mu}{h}_{\nu}\u27e9$) were corrected for their (trivially) expected correlation due to their mean activities (by subtraction of $\u27e8{v}_{i}\u27e9\u27e8{v}_{j}\u27e9$ and $\u27e8{h}_{\mu}\u27e9\u27e8{h}_{\nu}\u27e9$ respectively), so that only true correlations were assessed.
Calculating the normalized Root Mean Square Error
Request a detailed protocolGoodness of fit was quantified by computing the normalized Root Mean Square Error (nRMSE) for each statistic (shown in Figure 2—figure supplement 1). The RMSE between two vectors ${\text{x}}_{1},{\text{x}}_{2}$ of length ${N}_{x}$ is defined as $\text{RMSE}=\sqrt{\frac{1}{{N}_{x}}{\sum}_{n=1}^{{N}_{x}}{\left({x}_{1}(n){x}_{2}(n)\right)}^{2}}$. Ordinary RMSE was normalized so that different statistics could be compared, where 1 corresponds to ${\text{nRMSE}}_{\text{shuffled}}$, where both data and model statistics were randomly shuffled, and 0 corresponds to ${\text{nRMSE}}_{\text{optimal}}$ which is the RMSE between the training data and test data (by $\text{nRMSE}=1\frac{{\text{RMSE}}_{\text{ordinary}}{\text{RMSE}}_{\text{shuffled}}}{{\text{RMSE}}_{\text{optimal}}{\text{RMSE}}_{\text{shuffled}}}$).
Reconstruction quality
Request a detailed protocolAdditionally, we assessed the reconstruction quality of the test data. Here, the loglikelihood (LLH) between the test data $\mathbf{\text{v}}$ and its reconstruction $\mathrm{E}({\text{v}}_{\text{recon}})=\mathrm{E}(\text{v}\mathrm{E}(\text{h}\text{v}))\in [0,1]$ was computed. Because ${v}_{i}\in \{0,1\}$, the LLH is defined as
The resulting LLH was normalized (nLLH) such that 0 corresponds to an independent model (i.e., fitting neural activity with $\mathrm{E}({\text{v}}_{\text{recon}})(t)=\u27e8{v}_{i}\u27e9\forall t$) and 1 to optimal performance (which is ${\text{LLH}}_{\text{optimal}}=0$), by $\text{nLLH}=\frac{{\text{LLH}}_{\text{ordinary}}{\text{LLH}}_{\text{independent}}}{{\text{LLH}}_{\text{independent}}}$.
Generalized Linear Model
Request a detailed protocolWe used logistic regression, a Generalized Linear Model (GLM), to quantify the reconstruction quality of a fully connected model (i.e., with neurontoneuron connections, see Figure 2—figure supplement 2A). Logistic regression makes a probabilistic binary prediction (Bishop, 2006), hence allowing direct comparison to the probabilistic estimates of neural activity by the cRBM. In logistic regression, for a neuron ${v}_{i}(t)$ at time $t$, the activity of all other neurons ${\text{v}}_{i}(t)$ at time $t$ was used to predict ${\widehat{v}}_{i}(t)=P\left({v}_{i}(t)=1\right)=\frac{1}{1+\mathrm{exp}({\text{w}}_{i}\cdot {\text{v}}_{i}(t))}$ where ${\text{w}}_{i}$ is the estimated weight vector. This was implemented with scikitlearn (Pedregosa, 2011), using L_{2} regularization. L_{2} regularization was favored over L_{1} as it typically yields higher reconstruction performance; in the related context of protein contact map prediction, L_{2}regularized GLMs also better reconstructed contacts than L_{1}regularized GLMs (Morcos et al., 2011). The parameter ${\lambda}_{\text{GLM}}$ was optimized to ${\lambda}_{\text{GLM}}=1000$ using crossvalidation (Figure 2—figure supplement 2B). This is a computationally intensive model to compute because of the large number of regressor neurons $N1$: only $\sim 1000$ matrix rows could be inferred in 1 day on a 16thread CPU desktop computer. Therefore, we performed the crossvalidation of Figure 2—figure supplement 2B on 2% of all neurons (=1050 neurons) and computed the final distribution of Figure 2H on 10% of all neurons (=5252 neurons). GLMs were trained on the same train data as cRBMs, and evaluated on the same withheld test data as cRBMs (as described above).
Variational Autoencoders
Request a detailed protocolVariational Autoencoders (VAEs) were implemented in Tensorflow (2.1.10) using Keras (Chollet, 2015). For the encoder, we used a twolayer perceptron with intermediate layer size equal to the dimension of the latent space, a ReLU nonlinearity for the intermediate layer and no nonlinearity for the conditional mean and logvariance outputs. Batch normalization was used after each dense layer of the encoder. For the decoder, we used a dense layer with a sigmoid nonlinearity (no batch normalization). To obtain sparse assemblies, a L1 penalty on the weights of the decoder was added, such that the latent variables correspond to sparse neural assemblies at generation time. Models were trained by ELBO maximization using a very similar protocol as cRBMs: 200 K updates using the Adam optimizer (initial learning rate ${10}^{4}$, ${\beta}_{1}=0.9$, ${\beta}_{2}=0.999$ batch size: 100), with geometric decay schedule of learning rate after 50% of the training to a final learning rate of ${10}^{5}$. We tested the same hyperparameter range as for cRBMs, and selected the optimal model based on the heldout ELBO values (10 Gaussian samples per data configuration were used to compute the ELBO). The optimal hyperparameters were $M=300$, ${\lambda}_{1}=0.01$, but several values were very close to optimal (Figure 2—figure supplement 7A), including the value used for cRBMs ($M=200$, ${\lambda}_{1}=0.02$). We chose the latter for comparison to cRBM for the sake of simplicity, although we also included performance metrics of the $M=300,\lambda =0.01$ VAE model (Figure 2—figure supplement 7BH).
Regional occupancy
Request a detailed protocolWe determined the anatomical region labels of each neuron by registering our recordings to the ZBrain Atlas (as described previously). This yields a matrix $L$ of size ${N}_{\text{ZBA}}\times N$, which elements are ${l}_{r,i}=1$ if neuron $i$ is embedded in region $r$ and 0 if it is not. A cRBM neural assembly of HU μ is defined by its weight vector ${\text{w}}_{\mu}$ (of size $N$). Because cRBMs converge to sparse solutions, most of the weight elements will be very close to 0. To determine which anatomical regions are occupied by the assembly neurons with significantly nonzero weights, we computed the dot product between the weight vector ${\text{w}}_{\mu}$ and matrix $L$, leading to a weighted region label vector (of size ${N}_{\text{ZBA}}$) for each HU. The matrix of all $M$ weighted region label vectors is shown in Figure 2—figure supplement 6A for cRBM and Figure 2—figure supplement 6B for PCA.
The effective number of anatomical regions that one cRBM/PCA assembly is embedded in was then calculated using the Participation Ratio (PR) of each HU/Principal Axis. PRs are used to estimate the effective number of nonzero elements in a vector, without using a threshold (Tubiana and Monasson, 2017). The PR of a vector $\mathbf{\text{x}}=({x}_{1},\cdots ,{x}_{n})$ is defined by:
PR varies from $\frac{1}{n}$ when only 1 element of $\mathbf{\text{x}}$ is nonzero and $n$ when all elements are equal. We therefore estimated the effective number of regions by multiplying PR of the weighted region label vectors with the total number of regions ${N}_{\text{ZBA}}$ in Figure 2—figure supplement 6C.
Time constant calculation
Request a detailed protocolThe dReLU potential $\mathcal{U}}_{\mu$ of Equation 13 can learn to take a variety of shapes, including a doublewell potential (Tubiana et al., 2019a). HUs generally converged to this shape, giving rise to bimodal HU activity distributions (Figure 4). We determined the positions of the two peaks per HU using Gaussian Mixture Models fitted with two Gaussians. The bimodality transition point was then defined as the average between the two peaks (which was approximately 0 for most HUs). To calculate the time constant of state changes between the two activity modes, we subtracted the bimodality transition point from each HU activity ${h}_{\mu}$ individually. For clarity, all dynamic activity traces shown (e.g. Figure 4) are thus bimodality transition point subtracted. The time constant of an activity trace was then defined as the period of a (twostate) oscillation. A HU oscillation is defined as a consecutive negative and positive activity interval (because the bimodality now occurs at 0). A neuron oscillation is defined as a consecutive interspikeinterval and spikeinterval (which can last for multiple time steps, for example see Figure 1A, right panel).
Sorting of HUs
Request a detailed protocolHUs were sorted by hierarchical clustering of the Pearson correlation matrix of their dynamic activity (Figure 2B). Hierarchical clustering was performed using the Ward variance minimization algorithm that defines the distance between clusters (Virtanen et al., 2020). This sorting of HUs (and thus assemblies) is maintained throughout the manuscript for the sake of consistency.
Validating that the cRBM is in the compositional phase
Request a detailed protocolTo validate that the cRBMs converged to the compositional phase (see section  ‘Compositional restricted boltzmann machine’, compositional RBM formulation), we calculated the effective number of active HUs per data configuration (i.e., time step) $m(t)=\text{PR}\left({\text{h}}_{+}(t)\right)\cdot M$ where PR is the participation ratio (Equation 19), $M$ the number of HUs and ${h}_{+}=h{h}_{\text{inactive}}$, where ${h}_{\text{inactive}}$ is the inactive peak as calculated with the Gaussian Mixture Models (see section  ‘Time constant calculation’), because PR assumes that inactive elements are approximately zero (Tubiana et al., 2019a). A cRBM is said to be in the compositional phase if $1\ll \text{median}(m)\ll M$, which is true for all cRBMs (Figure 2—figure supplement 3).
Extensions of the structural connectivity matrix
Request a detailed protocolThe interregion structural connectivity matrix was derived from the single cell zebrafish brain atlas (Kunst et al., 2019). We used the postpublication updated data set from Kunst et al., 2019 (timestamp: 28 October 2019). The data set consists of $N=3098$ neurons, each characterized by the 3D coordinates of the soma center and of its neurites; there is no distinction between dendrites and axons. The brain is subdivided into $R=72$ regions and each neuron is duplicated by left/right hemisphere symmetry. We aim to estimate ${c}_{r,r\mathrm{\u2019}}$, the average strength of the connection between two neurons belonging to regions $r,{r}^{\prime}\in [1,R]$. For each neuron $n\in [1,N]$, we determine, using region masks, the region $r(n)$ where its soma is located and the cumulative length of the intersection between all its neurites and each region ${\mathrm{\ell}}_{n}(r)$. Under the assumptions that (i) the linear density of dendritic spines / axon presynaptic boutons is constant and (ii) the volumetric density of neurons is identical throughout regions, ${L}_{n}(r)$ is proportional to the volume ${V}_{r}$ of region $r$ times the average (bidirectional) connection strength between neuron $n$ and any neuron of region $r$. Aggregating over all neurons and symmetrizing, we obtain the following estimator for ${c}_{r,r\mathrm{\u2019}}$:
where ${\delta}_{r(n),r}=1$ if neuron $n$ has its soma in region $r$ and 0 if not. Using the same notations, the formula previously used in Kunst et al., 2019 is:
Equation 20 differs from Equation 21 in three aspects:
It discriminates between direct and indirect connections. Previously, a structural connection between region $r$ and region ${r}^{\prime}$ was established if a neuron had neurites with either tips or its soma within both regions. This may however result in indirect connections between $r$ and ${r}^{\prime}$, in cases where the neuron soma resides in another region ${r}^{\prime \prime}$. Here, we only account for direct connections, resulting in an overall slightly sparser connectivity matrix.
It is welldefined along the diagonal, i.e., for intraregion connections, whereas in Equation 21, each neurite would be counted as a selfconnection.
The denominator corrects for nonuniform sampling of the traced neurons throughout regions. Note that this issue only arose in the postpublication data set as nonuniform sampling was used to fill missing entries of the matrix.
Specimen averaging of connectivity matrices
Request a detailed protocolThe number of neurons in a particular brain region can vary across recordings from different specimen. Since the entries of the connectivity matrix are expected to be more accurate for wellsampled regions, we computed the weighted average of regiontoregion connections ${c}_{r,{r}^{\prime}}$ as follows:
Where ${T}^{F}$ is the recording length and ${N}_{{R}_{r}}^{F}$ is the number of neurons in region $r$ of fish $F$ that were recorded.
Correlation analysis of connectivity matrices
Request a detailed protocolPearson correlation was used to assess the similarity between cRBM functional connectivity matrices of different individual animals (Figure 5). Spearman correlation was used to compare structural connectivity versus functional connectivity (Figure 6), because these two metrics do not necessarily scale linearly. All correlation analyses, and the KilmogorovSmirnov test of Figure 6—figure supplement 1C, performed on symmetric matrices excluded one offdiagonal triangle (of symmetrical values) to avoid duplicates.
Data availability
The cRBM model has been developed in Python 3.7 and is available at: https://github.com/jertubiana/PGM (copy archived at swh:1:rev:caf1d9fc545120f7f1bc1420135f980d5fd6c1fe). An extensive example notebook that implements this model is also provided. Calcium imaging data preprocessing was performed in MATLAB (Mathworks) using previously published protocols and software (Panier et al., 2013; Wolf et al., 2017; Migault et al., 2018; Tubiana et al., 2020). The functional data recordings, the trained cRBM models and the structural and functional connectivity matrix are available at https://gin.gnode.org/vdplasthijs/cRBM_zebrafish_spontaneous_data. Figures of neural assemblies or neurons (Figure 1, 3) were made using the Fishualizer, which is a 4D (space + time) data visualization software package that we have previously published (Migault et al., 2018), available at https://bitbucket.org/benglitz/fishualizer_public . Minor updates were implemented to tailor the Fishualizer for viewing assemblies, which can be found at https://bitbucket.org/benglitz/fishualizer_public/src/assembly_viewer/. All other data analysis and visualization was performed in Python 3.7 using standard packages (numpy [Harris et al., 2020], scipy [Virtanen et al., 2020], scikitlearn [Pedregosa, 2011], matplotlib [Hunter, 2007], pandas [McKinney, 2010], seaborn [Waskom, 2021], h5py). The corresponding code is available at https://github.com/vdplasthijs/zfrbm (copy archived at swh:1:rev:b5df4e37434c0b18120485b8d856596db0b92444).

GINID cRBM_zebrafish_spontaneous_data. Data from: Neural assemblies uncovered by generative modeling explain wholebrain activity statistics and reflect structural connectivity.
References

Handbook of mathematical functions with formulas, graphs, and mathematical tablesAmerican Journal of Physics 56:958.https://doi.org/10.1119/1.15378

Largescale imaging in small brainsCurrent Opinion in Neurobiology 32:78–86.https://doi.org/10.1016/j.conb.2015.01.007

Habenula circuit development: past, present, and futureFrontiers in Neuroscience 6:51.https://doi.org/10.3389/fnins.2012.00051

Searching for collective behavior in a small brainPhysical Review. E 99:052418.https://doi.org/10.1103/PhysRevE.99.052418

Systematic errors in connectivity inferred from activity in strongly recurrent networksNature Neuroscience 23:1286–1296.https://doi.org/10.1038/s4159302006992

Bayesian inference of neuronal assembliesPLOS Computational Biology 15:e1007481.https://doi.org/10.1371/journal.pcbi.1007481

Random versus maximum entropy models of neural population activityPhysical Review. E 95:042321.https://doi.org/10.1103/PhysRevE.95.042321

Modeling the correlated activity of neural populations: a reviewNeural Computation 31:233–269.https://doi.org/10.1162/neco_a_01154

ConferenceRestricted boltzmann machines provide an accurate metric for retinal responses to visual stimuli5th International Conference on Learning Representations, ICLR 2017. pp. 24–26.

Neuronal assembliesIEEE Transactions on BioMedical Engineering 36:4–14.https://doi.org/10.1109/10.16444

Neural signatures of cell assembly organizationNature Reviews. Neuroscience 6:399–407.https://doi.org/10.1038/nrn1669

Training products of experts by minimizing contrastive divergenceNeural Computation 14:1771–1800.https://doi.org/10.1162/089976602760128018

Matplotlib: a 2D graphics environmentComputing in Science & Engineering 9:90–95.https://doi.org/10.1109/MCSE.2007.55

Information theory and statistical mechanicsPhysical Review 106:620–630.https://doi.org/10.1103/PhysRev.106.620

Modeling higherorder correlations within cortical microcolumnsPLOS Computational Biology 10:e1003684.https://doi.org/10.1371/journal.pcbi.1003684

ConferenceNumba: A llvmbased python jit compilerProceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC. pp. 1–6.

Detecting cell assemblies in large neuronal populationsJournal of Neuroscience Methods 220:149–166.https://doi.org/10.1016/j.jneumeth.2013.04.010

Development of oculomotor circuitry independent of hox3 genesNature Communications 5:1–12.https://doi.org/10.1038/ncomms5221

ConferenceData Structures for Statistical Computing in PythonPython in Science Conference. pp. 51–56.https://doi.org/10.25080/Majora92bf192200a

Redundancy and synergy of neuronal ensembles in motor cortexThe Journal of Neuroscience 25:4207–4216.https://doi.org/10.1523/JNEUROSCI.469704.2005

Cell assemblies in the cerebral cortexBiol Cybern 108:559–572.https://doi.org/10.1007/s0042201405964

Scikitlearn: machine learning in pythonThe Journal of Machine Learning Research 12:2825–2830.

Integration and multiplexing of positional and contextual information by the hippocampal networkPLOS Computational Biology 14:e1006320.https://doi.org/10.1371/journal.pcbi.1006320

WholeBrain activity mapping onto a zebrafish brain atlasNature Methods 12:1039–1046.https://doi.org/10.1038/nmeth.3581

Highdimensional ising model selection using ℓ1regularized logistic regressionThe Annals of Statistics 38:1287–1319.https://doi.org/10.1214/09AOS691

The structure of largescale synchronized firing in primate retinaThe Journal of Neuroscience 29:5022–5031.https://doi.org/10.1523/JNEUROSCI.518708.2009

BookInformation processing in dynamical systems: foundations of harmony theoryIn: Rumelhart DE, McClellan JL, editors. Parallel Distributed Processing Chapter 6. Cambridge: MIT Press. pp. 194–281.

ConferenceTraining restricted Boltzmann machines using approximations to the likelihood gradient25th international conference. pp. 1064–1071.https://doi.org/10.1145/1390156.1390290

ConferenceUsing fast weights to improve persistent contrastive divergenceProceedings of the 26th annual international conference on machine learning. pp. 1033–1040.

Emergence of spontaneous assembly activity in developing neural networks without afferent inputPLOS Computational Biology 14:e1006421.https://doi.org/10.1371/journal.pcbi.1006421

Modelbased decoupling of evoked and spontaneous neural activity in calcium imaging dataPLOS Computational Biology 16:e1008330.https://doi.org/10.1371/journal.pcbi.1008330

Emergence of compositional representations in restricted Boltzmann machinesPhysical Review Letters 118:138301.https://doi.org/10.1103/PhysRevLett.118.138301

BookRestricted Boltzmann Machines: From Compositional Representations to Protein Sequence AnalysisParis, France: PSL Research University.

Blind deconvolution for spike inference from fluorescence recordingsJournal of Neuroscience Methods 342:108763.https://doi.org/10.1016/j.jneumeth.2020.108763

SoftwareProbabilistic graphical models (PGM, version swh:1:rev:caf1d9fc545120f7f1bc1420135f980d5fd6c1feSoftware Heritage.

Integrative wholebrain neuroscience in larval zebrafishCurrent Opinion in Neurobiology 50:136–145.https://doi.org/10.1016/j.conb.2018.02.004

LightSheet functional imaging in fictively behaving zebrafishNature Methods 11:883–884.https://doi.org/10.1038/nmeth.3040

Seaborn: statistical data visualizationJournal of Open Source Software 6:3021.https://doi.org/10.21105/joss.03021

The structure of the nervous system of the nematode Caenorhabditis elegansPhilosophical Transactions of the Royal Society of London. Series B, Biological Sciences 314:1–340.https://doi.org/10.1098/rstb.1986.0056

Sensorimotor computation underlying phototaxis in zebrafishNature Communications 8:651.https://doi.org/10.1038/s41467017003103
Article and author information
Author details
Funding
Biotechnology and Biological Sciences Research Council (BB/M011224/1)
 Thijs L van der Plas
Edmond J. Safra Center for Bioinformatics at Tel Aviv University
 Jérôme Tubiana
Human Frontier Science Program (LT001058/2019C)
 Jérôme Tubiana
European Research Council (715980)
 Volker Bormuth
Human Frontier Science Program (RGP0060/2017)
 Georges Debrégeas
Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016.VIDI.189.052)
 Bernhard Englitz
 Thijs L van der Plas
The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.
Acknowledgements
TLvdP had an Erasmus +fellowship (European Union) and acknowledges support from the Biotechnology and Biological Sciences Research Council (BBSRC, grant No. BB/M011224/1). JT acknowledges support from the Edmond J Safra Center for Bioinformatics at Tel Aviv University and from the Human Frontier Science Program (crossdisciplinary postdoctoral fellowship LT001058/2019 C). GM was funded by a PhD fellowship from the Doctoral School in Physics, Ile de France (EDPIF). GLG had a PhD fellowship from the Systems Biology Network of Sorbonne Université. BE and TLvdP. were supported by an NWOVIDI Grant. Funding sources: European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation program grant agreement number 715980. Human Frontier Science Program (RGP0060/2017). The French Research National Agency under grant No. ANR16CE160017. Dutch Institute for Scientific Research NWO (Nederlandse Organisatie voor Wetenschappelijk Onderzoek) grant No. 016.VIDI.189.052. We thank the IBPS fish facility staff for the fish maintenance, in particular Stéphane Tronche and Alex Bois. We are grateful to Carounagarane Dore for his contribution to the design of the experimental setup. We thank Misha Ahrens for providing the GCaMP lines. We are also grateful to Rémi Monasson for very fruitful discussions and his comments on the manuscript.
Ethics
Experiments were approved by Le Comité d’Éthique pour l’Experimentation Animale Charles Darwin C2EA05 (02601.01).
Version history
 Preprint posted: November 11, 2021 (view preprint)
 Received: September 21, 2022
 Accepted: January 15, 2023
 Accepted Manuscript published: January 17, 2023 (version 1)
 Version of Record published: February 20, 2023 (version 2)
Copyright
© 2023, van der Plas, Tubiana et al.
This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.
Metrics

 1,781
 views

 262
 downloads

 10
 citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Further reading

 Neuroscience
One wellknown biomarker candidate that supposedly helps capture fluid cognition is Brain Age, or a predicted value based on machinelearning models built to predict chronological age from brain MRI. To formally evaluate the utility of Brain Age for capturing fluid cognition, we built 26 ageprediction models for Brain Age based on different combinations of MRI modalities, using the Human Connectome Project in Aging (n=504, 36–100 years old). First, based on commonality analyses, we found a large overlap between Brain Age and chronological age: Brain Age could uniquely add only around 1.6% in explaining variation in fluid cognition over and above chronological age. Second, the ageprediction models that performed better at predicting chronological age did NOT necessarily create better Brain Age for capturing fluid cognition over and above chronological age. Instead, betterperforming ageprediction models created Brain Age that overlapped larger with chronological age, up to around 29% out of 32%, in explaining fluid cognition. Third, Brain Age missed around 11% of the total variation in fluid cognition that could have been explained by the brain variation. That is, directly predicting fluid cognition from brain MRI data (instead of relying on Brain Age and chronological age) could lead to around a 1/3time improvement of the total variation explained. Accordingly, we demonstrated the limited utility of Brain Age as a biomarker for fluid cognition and made some suggestions to ensure the utility of Brain Age in explaining fluid cognition and other phenotypes of interest.

 Developmental Biology
 Neuroscience
Movement is a key feature of animal systems, yet its embryonic origins are not fully understood. Here, we investigate the genetic basis underlying the embryonic onset of movement in Drosophila focusing on the role played by small noncoding RNAs (microRNAs, miRNAs). To this end, we first develop a quantitative behavioural pipeline capable of tracking embryonic movement in large populations of fly embryos, and using this system, discover that the Drosophila miRNA miR2b1 plays a role in the emergence of movement. Through the combination of spectral analysis of embryonic motor patterns, cell sorting and RNA in situs, genetic reconstitution tests, and neural optical imaging we define that miR2b1 influences the emergence of embryonic movement by exerting actions in the developing nervous system. Furthermore, through the combination of bioinformatics coupled to genetic manipulation of miRNA expression and phenocopy tests we identify a previously uncharacterised (but evolutionarily conserved) chloride channel encoding gene – which we term Movement Modulator (Motor) – as a genetic target that mechanistically links miR2b1 to the onset of movement. Cellspecific genetic reconstitution of miR2b1 expression in a null miRNA mutant background, followed by behavioural assays and target gene analyses, suggest that miR2b1 affects the emergence of movement through effects in sensory elements of the embryonic circuitry, rather than in the motor domain. Our work thus reports the first miRNA system capable of regulating embryonic movement, suggesting that other miRNAs are likely to play a role in this key developmental process in Drosophila as well as in other species.