Introduction

A central goal of neuroscience is to understand how the structure and dynamics of neural activity relate to the internal states of the organism and the external world. This understanding is derived from analyzing complex, multi-modal datasets. While the community has improved tools and algorithms for data collection and analysis^1–6, extracting consistent and reproducible insights from data remains a complex and time-consuming task. Often, researchers take years to collect and organize data, which is then transformed through a complicated series of analyses using custom scripts. This begins with preprocessing that isolates specific signals from the data, followed by multiple subsequent analyses that quantify properties of these signals. The outputs of these analyses are then synthesized across datasets, and when they are consistent upon limited replication, they are reported in the scientific literature—with the data and analysis scripts documented to varying degrees.

Ideally, it would be possible for another group to take the same raw datasets, apply the analyses, and rapidly and reliably reproduce the findings. In practice, this is often difficult, if not impossible. Raw data are seldom shared and metadata critical for understanding the data are often not included, posing a significant challenge to the replication process. Essential components of the analysis pipelines, such as the manual curation of clustered spikes and artifact rejection, are often irretrievable from the written reports. Similarly, the full set of parameters used for each of the analyses are not shared or hidden in cryptic analysis scripts. Efforts to reproduce findings are also hampered by idiosyncratic data and code organization, poor documentation, and missing vital details, including computational hardware requirements⁷. In collaborations among multiple scientists, these problems are exacerbated due to the variability in how each participant carries out analysis. Consequently, the full validation of a result usually requires repeating the experiment and reconstructing the analysis from scratch.

These difficulties in replication of analyses incur significant costs in time and effort. A new trainee might struggle to analyze existing data because they do not understand critical details. A scientist who downloads the data from a previous study may find that the analyses they wanted to carry out are impossible because the raw data is not available. Alternatively, raw data may be available, but the scientist may need intermediate results (e.g. spike waveforms) that are not included. Similarly, shared code, including visualizations, is most often not standardized or documented, causing multiple teams to duplicate efforts and implement the same tools.

A system that addresses these challenges therefore should enable:

• compilation of raw data with sufficient metadata for analysis and reuse

• sharing of data and all intermediate analysis results in an accessible format

• reproducible analysis via well-documented, organized, and searchable pipelines

• generation of shareable visualizations to facilitate communication and collaboration

• easy use by scientists with minimal formal training in data management.

Achieving these goals would represent a major step towards meeting the FAIR guiding principles for findable, accessible, interoperable, and reusable⁸ data and analysis pipelines⁹. For example, it would become possible to easily find publicly available data, analyze it with a standardized pipeline that keeps track of all the parameters, and generate a visualization to share the results over the web—a stark contrast to how science is practiced today.

In pursuit of this vision, many organizations, such as the Allen Institute for Brain Science (AIBS), Johns Hopkins Applied Physics Lab (APL), and the International Brain Laboratory (IBL), have made strides by standardizing and sharing data and analyses^10–12. However, these efforts have not fully resolved the issues related to data sharing and reproducible analysis. For instance, a lack of raw data often precludes reproduction of early stages of the analysis, and many steps of the processing pipeline (e.g. the criteria used for manual or automatic curation of spike sorting) that can significantly affect the results¹³ are omitted. Some of the technologies used, such as cloud computing services (e.g., CodeOcean) and sophisticated databases and APIs^11,14,15, can be cost prohibitive or require specialized software engineering expertise that is beyond the reach of most labs. Furthermore, these existing efforts tend to be focused on the needs of specific projects, data types, and behavioral paradigms, limiting their scope. Thus, while these efforts mark important advances, there remains a need for user-friendly, integrated solutions that can be widely adopted across individual labs in the neuroscience community.

To address this need, we developed Spyglass, an open-source neuroscience data management and analysis framework written in Python. Spyglass builds on widely available community-developed tools and adopts the Neurodata Without Borders (NWB) as the standardized format^16,17. It uses DataJoint^5,18 to manage reproducible analysis pipelines with a relational database and incorporates novel software tools (e.g. Kachery and Figurl) for sharing data and web-based visualizations to enable collaboration within and across labs. This includes methods for exporting and uploading all raw data and intermediate results used to produce a manuscript, which, along with sharing of code, enables full replication of results. Spyglass is Python-based and thus can accommodate pipelines that use a wide array of analysis packages that have been developed by the community, including SpikeInterface¹⁹, GhostiPy²⁰, DeepLabCut², and Pynapple²¹. Spyglass also offers ready-to-use pipelines for analyzing behavior and electrophysiological data, including spectral analysis of local field potential (LFP), spike sorting, video processing to extract position, and decoding neural data. Spyglass can be extended to support additional pipelines for behavioral, intracellular, optical physiology data, or other data types that can be stored in the NWB format. In addition to extensive documentation and tutorials, new users can try out a demo version of Spyglass hosted on the web by HHMI and 2i2c as a Jupyter Hub instance. Here we describe the structure of Spyglass and demonstrate its potential by applying the same analysis pipelines to NWB files from different labs and comparing the results.

Results

Overview of Spyglass

Spyglass is an open-source Python-based software framework for reproducible analysis of neuroscience data and sharing of the results with collaborators and the broader community (Figure 1). It can be run locally or in the cloud. Analyzing data with Spyglass begins with raw data and experimental metadata stored in the NWB format^16,22. These NWB files are ingested into a relational database and processed using DataJoint-enabled pipelines. These pipelines are built around common neuroscience tasks such as spectral filtering, spike sorting, pose tracking, and neural decoding. DataJoint stores parameters of each analysis and tracks the intermediate analysis results, which are also stored as NWB files to maintain a shareable standardized data format. Spyglass enables sharing results and interactive visualization of the data over the web via Kachery and Figurl. Finally, Spyglass supports exporting specific parts of the database required to reproduce the results and figures of a manuscript and the upload of the associated raw data and analysis outputs to a public repository. In the following sections, we provide detailed descriptions of these components and the design decisions behind them.

Figure 1:

The NWB format as data specification

Why NWB?

A typical neuroscience experiment consists of multiple data streams stored in different formats. Managing such heterogeneous data in a shareable and accessible manner is challenging. A practical solution is to save the data in a community-supported format like NWB, which is emerging as a standard for neurophysiology and behavior data^16,22. We have chosen NWB as the data specification in Spyglass for the following reasons:

• The versatility of NWB accommodates various data types and allows metadata to be saved with the data in a single self-annotated file.

• NWB files are immediately shareable.

• Public data archives like DANDI^23–26 accept the NWB format and provide APIs to easily stream file contents for local analysis.

• Tools developed for NWB files are immediately accessible to users.

Conversion to NWB can be done using software tools developed by the community, such as the NeuroConv package or NWB GUIDE, a desktop app for converting data to NWB without having to write code.

Importantly, Spyglass requires all raw data—including neurophysiology, behavioral task, interaction with the environment— to be in the NWB format prior to any analysis. This ensures reproducibility of all subsequent analyses by sharing the NWB file containing the raw data and the analysis pipelines. Furthermore, Spyglass stores virtually all intermediate results from downstream analysis pipelines in NWB. This ensures that all data associated with the analysis can be shared and read using the same software tools.

Spyglass-specific NWB requirements

NWB allows some flexibility in the specification of data to accommodate a broad range of experiments and lab-specific requirements. For example, the name of data types within the NWB file can differ from those expected by Spyglass. We have fully described the Spyglass-specific NWB conventions in the documentation website. To further accommodate NWB files from many sources, we have also developed a system that makes it possible to ingest NWB files into Spyglass even when they do not adhere to our naming conventions or best practices by including a configuration yaml file (see Methods and Table 1, 02_Insert_data).

Table 1:

Relational database as analysis pipelines

Why a relational database?

One significant challenge with data analysis is in managing its complexity. Most results derive from an extended series of steps, including “preprocessing” (e.g. spike sorting for electrophysiological data, region-of-interest identification for optical physiological data, video processing for behavioral data, etc.) and downstream analyses. Each step depends on a different algorithm with a specific set of parameters and generates distinct intermediate data. Tracking these numerous components is difficult, and understanding how another scientist has managed them can be even more daunting. This complexity hinders collaboration, verification of results, and data reuse.

These issues motivated our use of a formal software system: the relational database, a well-known data structure that uses tables to organize data. To construct an analysis pipeline, we use DataJoint^27,28 to define a series of database tables with a dependency structure. Associated with the database tables is code that carries out an analysis using a specific set of parameters and a specific part of the data. This code then stores the results as a new row in the table with a pointer to the results stored on disk as an NWB file. Creating a new table row is referred to as “populating” the table. Thus, data analysis becomes a matter of populating and interacting with the database. This style of data analysis offers many advantages:

• It makes it easy to apply the same analysis to multiple datasets, as users need to specify only the data and parameters for computation (“what”) rather than the execution details (“how”).

• It provides a structure to organize the analysis parameters, data, and outputs into different tables.

• It enables easy access to multiple datasets via queries.

• It is concurrently accessible to multiple users.

Because DataJoint binds the code for running the computation with the table that will store the result, populating the same table will execute the same code. DataJoint also provides additional features for reproducible data analysis, such as maintaining data integrity of the database (e.g. deleting a table entry causes cascading deletion of dependent entries in downstream tables) and the files containing the results (e.g. by checksum verification).

How does Spyglass differ from DataJoint?

While Spyglass is based on DataJoint, it offers many useful features that DataJoint lacks. These include:

• A tight integration with the NWB format: When the NWB file is ingested into Spyglass, pointers to the data types appear as rows in a set of predefined tables. These serve as the starting point for analyses and an interface for the users to access the raw data within Spyglass. We provide the mapping between NWB data types and corresponding Spyglass tables in the documentation website.

• Extending table types: Spyglass provides a mix-in class for defining table types that are not included in DataJoint or extending the function of existing table types. This is used to implement many key table types such as Merge tables, which allow multiple upstream pipelines to feed into the same downstream pipeline. This example is illustrated in the description of the spike sorting pipeline below.

• Permission-based delete: Spyglass enables the deletion of individual rows in a table based on pre-defined user permission. This is not naturally supported by MySQL, the underlying relational database management system used by DataJoint.

• Improved searching based on restrictions on non-primary keys: Spyglass allows the users to conveniently track the provenance of a particular row in a downstream table across multiple upstream tables with only partial information.

• Export system for publishing: Spyglass provides a convenient way to export only the part of the database used for generating results and figures for a publication. This is done by caching the information about tables that are accessed when generating figures.

• The inclusion of various helper functions, which are detailed on the Spyglass documentation website.

Setup and hardware requirement

Spyglass can be installed in any setting that can support Python via the Python Package Index (PyPI). We provide detailed installation instructions on the documentation website, including a complete list of software requirements. In addition to the Python package, using Spyglass requires running a relational database (currently MySQL backend is supported). In our laboratory, we run this from a Docker container provided by DataJoint on a lab-wide server and grant access to it to members of the lab and other collaborators. This local configuration is recommended for use cases involving ∼ 10 users. For a larger scale deployment, one could also run the Spyglass database in the cloud using services such as AWS.

Practical use cases and extensions

Spyglass comes with many pre-defined pipelines that implement common analysis tasks for electrophysiological and behavioral data. For users interested in using these pipelines, they can do so as soon as they ingest their NWB files into the database. Spyglass can also serve as a jumping off point for exploratory data analysis. For example, the user can conveniently read specific data types from the NWB file by first ingesting it into Spyglass and accessing database tables with Spyglass functions (e.g. fetch_nwb). If they need to pre-process the data first, they can do by running the relevant pipelines. Once the user has decided to formalize a particular analysis that is not yet supported by Spyglass, they can extend Spyglass and create user-generated custom pipelines. These could include data types from NWB files not currently supported by Spyglass (e.g. photometry, optical physiology, etc.), or build on existing Spyglass pipelines. Because the raw data and intermediate results are in NWB format, the custom pipelines can take advantage of analysis software packages within the NWB ecosystem.

Organization of analysis pipelines

Here we delve deeper into the design and organization of analysis pipelines in Spyglass. As mentioned previously, the analysis pipelines are defined as a set of tables in the relational database. Specifically, Spyglass uses DataJoint syntax to define tables as Python classes. The code for executing the analysis is associated with these tables as class methods, enabling a tight integration of the database structure with the code for populating it. We refer the reader to the DataJoint documentation for more details on specific commands to interact with the database.

When an NWB file is first ingested into Spyglass, pointers to the data types in the NWB file are stored in database tables of the Common module. Each Common table corresponds to a data object in the NWB file and serves as an interface to retrieve it with simple function calls (fetch_nwb). The retrieval is “lazy” in the sense that only a specific part of the data is loaded for analysis instead of the entire NWB file.

An analysis pipeline consists of sets of tables downstream of the Common tables. In each step in the analysis, the user populates one of four table types (Figure 2A):

• Data tables contain pointers to data objects in either the original NWB file or ones generated by an upstream analysis.

• Parameter tables contain a list of the parameters needed to fully specify the desired analysis.

• Selection tables associate parameter entries with data object entries.

• Compute tables execute the computations to carry out the analysis using the Data and Parameters specified in the Selection table entry. These results are then stored and can serve as Data for downstream analysis.

Figure 2:

Spyglass includes pipelines for standard analysis tasks in systems neuroscience, such as the analysis of LFP, spike sorting, video and position processing, and fitting state-space models for decoding neural data. Tutorials for all pipelines are available on the Spyglass documentation website (Table 1). Many pipelines are powered by community-developed, open-source packages, like GhostiPy²⁰, SpikeInterface¹⁹, DeepLabCut² and Moseq²⁹. These pipelines store a complete record of the analysis and simplify the application of these tools. Furthermore, multiple versions of the pipelines can co-exist to apply different algorithms to a single data set, making it easy to probe the robustness of the results (see Merge motif below).

Next we provide a detailed description about the implementation of three common analysis tasks in Spyglass pipelines: (i) filtering broadband extracellular voltage traces to extract the lower-frequency LFP bands; (ii) detecting discrete events (e.g. sharp-wave ripples, a hippocampal event marking the time of bursts of population activity) in the LFP signals; and (iii) spike sorting and curation.

Example 1: LFP extraction (Figure 2B)

To extract the LFP signal (below 400 Hz), we use the pipeline shown in Figure 2B. First, we select a row from the Raw table, a Data table that points to an ElectricalSeries object in the NWB file. We then specify the parameters of the analysis in the Parameter tables: the list of channels for which LFP should be extracted (LFPElectrodeGroup), the time interval for the LFP extraction (IntervalList), and the coefficients for the filter that will be used on the data (FIRFilterParameters). These parameters are associated with the entry in the Raw table by defining a Python dictionary object that specifies the Data and Parameter entries and inserting it into a Selection table (LFPSelection) by calling the LFPSelection.insert1 method (Figure 2B). Finally, we apply the filter to the selected data over the selected interval using the LFP table (a Compute table) by calling the LFP.populate method. The resulting filtered data is saved to disk in the NWB format, and the object ID associated with the LFP object within the NWB file is also stored in the LFP table for easy retrieval. Thus, the corresponding entry in the LFP table contains all the details about the data and the parameters, allowing a user to fully track the provenance of the output.

Example 2: Sharp-wave ripple detection (Figure 2C)

Once the LFP extraction is completed, we can build on the results by applying another filter to isolate a specific frequency band and identifying sharp-wave ripples (SWRs), a prominent LFP event within hippocampal data. This pipeline is illustrated in Figure 2C. It applies two additional steps to a row in the LFP table: another band-pass filter to isolate the 150-250 Hz band and a subsequent detection of SWR events. Each step uses the same basic scheme shown in Figure 2A. These include defining a specific band-pass filter in the Parameter tables; selecting a time interval for the bandpass filtering; and adding an entry to LFPBandSelection table that binds both the filter parameters and the time interval with a row in the LFP table. A call to LFPBand.populate generates an NWB file containing the ripple-band data and an entry in the LFPBand table with information about which data and parameters were used. Next, the user selects an entry in RippleParameters to define the parameters for detecting the ripple events (e.g. threshold over the spectral power) and associates it with filtered data in LFPBand in the RippleLFPSelection table. Finally, the RippleTimes table is populated (by RippleTimes.populate), which identifies the start and end times of each ripple event and saves these to a new NWB file.

Example 3: Spike sorting and curation (Figure 3)

The spike sorting pipeline (Figure 3) combines the principles of analysis pipeline design we outlined previously with additional design features. This pipeline includes the following steps: (1) preprocess the recording (e.g. filter and whiten to remove noise); (2) apply spike sorting algorithm (e.g. MountainSort4, Kilosort3, etc.); (3) curate the results (e.g. either manually or automatically by computing quality metrics); and (4) consolidate the output with other sources of sorted units (e.g. those already present in the NWB file) for downstream analysis. Each of these steps follow the general design shown in Figure 2A. We also detail additional features that have not been discussed previously.

Figure 3:

Global Parameter tables (e.g. IntervalList)

An important object in any analysis is the time interval during which the data were collected or to which analysis procedures should be applied. To avoid having a separate table for time intervals in every pipeline, we store them in the IntervalList table of the Common module for all pipelines. For example, in the spike sorting pipeline (Figure 3), IntervalList provides a time interval for both preprocessing the recording (SpikeSortingRecordingSelection) and running a spike sorting algorithm (SpikeSortingSelection). In addition, the intervals during which artifacts (i.e. high-amplitude voltage transients from behavioral events such as licking) occur is identified and fed back into IntervalList (dashed arrow in Figure 3).

“Cyclic iteration” motif for curation

Certain pipelines, such as curating the output of spike sorting, may need to be run multiple times on the same data. For example, one might first compute quality metrics to identify noise clusters and potential candidates for merging over-clustered units (Automatic); then inspect, merge, and apply curation labels to the result with an external viewer (Manual); and finally, compute a final set of metrics to describe the quality of each unit (Automatic). This results in a sequence of curation steps: Automatic, Manual, Automatic. Depending on the data, the user may choose a different curation sequence, and the order and length of these sequences might change as new algorithms and metrics are developed. This presents a challenge in modeling the pipeline within the relational database.

We therefore developed a specific design motif to enable this iterative curation with a finite number of tables (Figure 3). First, a given row of the CurationV1 table (the output of the spike sorting step) is taken through automatic or manual curation steps downstream. Upon completion, the spike sorting object may enter this curation pipeline again as a new row in the CurationV1 table. Importantly, the new row has information about previous curation from which it descended. This allows the user to track each round of curation while applying as many steps as desired. It can also be easily extended; if new automatic curation algorithms are developed in the future, it can simply be added downstream to the CurationV1 table, enabling application of the latest methods to previously collected data.

“Merge” motif for consolidating data streams and versioning pipelines

A different challenge arises when the user wants to feed multiple streams of data of the same type into a single downstream pipeline. For example, once curation is completed, the spike sorting is saved in CurationV1. But some NWB files may already contain curated spike sorting (as a row in the table ImportedSpikeSorting), and one may want to apply the same downstream pipeline to both data sources to compare the results. In yet another case, the other data stream could be a different version of the spike sorting pipeline (e.g. CurationV2) that uses different algorithms but produces output of the same type. Adding the same downstream pipeline to each of these separately would result in code redundancy and database bloat. Simply having these converge onto a single downstream table is not desirable either, as it will require modifying an existing table to add new columns every time a new version or new data stream is added.

To solve this problem, we have designed a “merge” table type (Figure 3). Here Parts tables (a table type within DataJoint tightly associated with a parent table) are used to implement the merging of multiple data streams onto a single table. The downstream pipeline then gets data from this table without any duplication. More details for the implementation and helper functions to maintain data integrity can be found in the tutorial notebook (Table 1, 04_Merge_Tables).

Sharing Data, Analysis, and Visualization

Complete sharing of data and analysis at the end of projects

A key goal of our system is to simplify sharing data and analyses when results are ready to be published. Because all raw and intermediate data are in the NWB format, they can be directly deposited to DANDI^16,24–26, a NIH-supported public archive for neuroscience data. Sharing the analysis code is also easy: simply share the codebase for the analysis pipelines (i.e. Spyglass plus any project-specific pipelines) and the scripts used to populate the database. Others can then download the raw data from DANDI, set up the database with Spyglass, and recreate all results locally by executing the population script. Alternatively, users may want to share the Spyglass database in its populated state so that the community can access it directly without going through the setup procedures or re-running time-consuming analysis steps. This can be done by (i) hosting the database on the cloud and granting access to users outside the lab; or (ii) exporting and sharing parts of the database that were used by the project. Spyglass facilitates the second option by providing functions that will automatically log the tables and NWB files used for creating figures of a manuscript in a Python environment (Table 1, 05_Export). These are saved as Docker containers and optionally uploaded to DANDI.

Controlled sharing for ongoing projects

For ongoing projects, users may want to limit the sharing of the analyses to their collaborators. This requires controlling access to the database and the underlying NWB files that contain the raw or intermediate data. This is straightforward to manage in Spyglass. DataJoint handles access to the database natively by requiring a username and a password. Managing access to the NWB files is handled by Kachery, a content-addressed sharing tool for scientific data (Figure 4A). Specifically, the user selects the NWB files to be shared by inserting pointers to them into NwbKachery and AnalysisNwbKachery tables within Spyglass. When the collaborator attempts to access these files, Kachery first looks for them in their local system. If not found, the corresponding files are automatically uploaded from the user’s system to a cloud storage server and then downloaded to the collaborator’s computer. This feature is detailed in a tutorial (Table 1, 03_Data_Sync). Kachery offers advantages over other file hosting services (e.g. Dropbox and Google Drive) or alternative architectures (e.g. IBL data architecture) by not requiring a central location to track available files and providing a user-friendly Python API.

Figure 4:

Sharing visualizations

Spyglass enables users to create and share interactive visualizations of final and intermediate analysis results through the Figurl package. These visualizations facilitate understanding complex, multi-modal neuroscience datasets by allowing users to (i) quickly compare different stages of processed data to spot issues with their data and (ii) align multimodal information sources to get a more holistic view of their dataset. Figurl is integrated within Spyglass as dedicated tables attached to specific pipelines such as spike sorting (Figure 4B) and neural decoding (Figure 5). Populating these tables generates a URL to web-based visualizations for exploring complex, multi-dimensional time series across multiple views whose time axes can be linked. Sharing them is also easy, as the URL can be accessed from any browser without the need for local software installation or specialized hardware. This allows collaborators anywhere in the world to easily access and explore the data.

Figure 5:

Demonstration of generalizability: neural decoding of position in multiple data sets

A major goal of Spyglass is to facilitate the analysis of data across multiple datasets that may come from different laboratories. To illustrate this, we ingested and analyzed two NWB files containing single-neuron recordings from rat hippocampus, one from our laboratory and another from the Buzsáki laboratory at NYU³⁰. Specifically, we applied a switching state space model ^31,32 to decode the animal’s position from spikes and infer periods of different types of non-local representations (such as replay and theta sequences), during which the decoded position deviates from the animal’s true position. This is a complex analysis that involves integrating multiple data sources, including position and neural spiking activity, and applying an advanced statistical model with many user-defined parameters. The decoding pipeline in Spyglass enables the user to carry out every step of this analysis, including “preprocessing” of the data (e.g. linearize the 2D position of the animal, perform spike sorting, or import units that have already been sorted) and fitting of the model. After running the decoding pipeline, we visualize the results on the browser via Figurl and generate plots to further reproduce the reported results.

The UCSF dataset contains large scale hippocampal recordings in an animal performing a foraging task in a maze with six reward sites and dynamic reward probabilities (Figure 5A, top panel). Applying the decoding pipeline to these data yields a probability distribution over space in 2 ms bins that describes our estimate of the “mental” position of the animal. This mental position tracks the animal as it traverses the maze (Figure 5A, 2^nd panel from top; see interactive visualization via Figurl) but also shows interesting systematic deviations from actual position. Computing the distance between the peak of the probability distribution and the actual location reveals characteristic patterns of such deviations from the actual position (Figure 5A, 3^rd panel from top) in which the decoded position sweeps ahead of the actual position and then back during movement bouts. This pattern recurs at ∼8 Hz, reflecting the well-known “theta sequences” seen in the hippocampus^33,34.

We then applied this same pipeline to the NYU dataset, where animals performed a spatial alternation task on a maze with a figure-8 topology (Figure 5B, top panel). As expected, we could identify theta sequences in these data as well, highlighting the robustness of these phenomena (Figure 5B, 2^nd and 3^rd panels from top, see interactive visualization via Figurl). Moreover, the NYU dataset includes a specific manipulation in which the medial septum, a brain region critical for pacing the theta rhythm, was cooled, reducing the theta frequency from 8-10 Hz to 5-8 Hz. The authors originally carried out several detailed analyses to demonstrate that cooling reduced theta frequency and impaired behavior without changing the overall spatial tuning of single neurons or their tendency to fire sequentially within theta cycles. However, the authors did not apply state-space decoding methods, and did not characterize the effects of cooling on the decoded representation of space in relation to the animal’s actual position. We therefore applied our decoding pipeline to the cooling trials (“cooling on”) and the control trials preceding it (“pre-cooling”), just after it (“cooling off”), and the recovery trials 10-12 minutes after cooling (“post-cooling”).

The results of these analyses were consistent with the published findings and provided new characterizations that could serve as the foundation for additional discoveries. We first estimated the multiunit firing rate as a proxy for the theta LFP and characterized its power spectrum before and after cooling. As expected, cooling decreased the power above ∼8 Hz and increased the power below ∼8 Hz, consistent with the slowing of theta LFP shown in the original manuscript (Figure 5C, top panel). We then applied the same analysis described above to the distance between the decoded and the actual position during movement (“decode distance”), expecting cooling to have a similar effect on its power spectrum. Interestingly, here cooling led to a decrease in power at essentially all frequencies (Figure 5C, bottom panel). Consistent with this result, the decode distance decreased from the pre-cooling to cooling period, with a partial recovery during the post-cooling period (Figure 5D, top panel). Similarly, the average speed at which the decoded position moved ahead and behind the animal was also reduced during cooling and showed a partial recovery after the cooling period (Figure 5D, bottom panel). These results indicate that cooling reduces both the extent and the rate at which the decoded position deviates from the actual position. This was unexpected given that cooling had no effect on the average spatial tuning of these cells³⁰. It also raises an interesting hypothesis: hippocampal representations of distant locations may be exquisitely tuned to the specific frequency of the rhythmic input from medial septum, such that slowing the rhythm down by just 2-3 Hz significantly limits their expression. More broadly, these findings illustrate the power of our framework that enables both replication of results across datasets and the re-analysis of previously collected data.

Discussion

Summary of results

Science is a social enterprise that relies heavily on collaboration and transparency among researchers^35,36. Reproducible and shareable data analysis plays a critical role in this context, as it ensures that scientific findings can be independently verified and built upon by others. To facilitate this for the neuroscience community, we built Spyglass, a software framework that combines the NWB format and the relational database structure. Building on many community-developed tools, it provides useful features to design complex analysis pipelines, share raw and processed data, generate web-based visualizations, and analyze data from multiple sources. As a result, it simplifies collaboration within and across labs, making it well-suited as a community framework for neurophysiological and behavioral data analysis.

Comparison to prior work

Our work builds on many previous approaches that have been proposed for scientific data management and reproducible analysis pipelines. This includes work from individual laboratories that have illustrated how a few elements of an NWB file could be read into a DataJoint database, and publications highlighting datasets available in NWB³⁷. More broadly, DataJoint is used by many labs with lab-specific pipelines³⁸, but to our knowledge none of these efforts integrate cross-laboratory data and visualization tools or use NWB as the foundation to facilitate sharing. Our system also contains elements similar to those developed by large collaborative groups like The International Brain Laboratory (IBL) that are designed to organize neurophysiology data for sharing with collaborators and a module to automatically run analyses¹². But the conversion to a standardized format (outside the collaboration or group) and public data sharing are only done following substantial analysis in the IBL system, complicating replication of the full analysis.

Other approaches do away with the relational database altogether. For example, DataLad uses version control tools such as git and git-annex to manage both code and data as files³⁹. This enables the creation of a data analysis environment and decentralized data sharing. For building analysis pipelines, it may be combined with other tools for managing the sequential execution of scripts. For example, Snakemake⁴⁰ (and related projects such as Cobrawap⁴¹) allows the users to gather and define the input, output, and the associated scripts to execute for each analysis step, thereby tracking the dependency between steps. But because these tools do not provide any formal structure for data analysis, they lack the advantages of the relational database that we discussed, such as being able to easily organize or search for the records of previous analysis based on specific parameters, efficient data sharing and access management to multiple users, and built-in data integrity checks based on constraints native to the database (e.g. primary keys).

By contrast, Spyglass begins with a shared data format that includes the raw data and offers both transparent data management and reproducible analysis pipelines using a formal data structure. One distinguishing feature of Spyglass is the emphasis placed on combinatorial matching of data and method in a reproducible way. For example, Spyglass makes it easy to apply multiple spike sorting algorithms to a given dataset and to compare the results, as this involves simply matching the data with different rows in the parameter tables. Spyglass also makes it straightforward to apply complex analyses like decoding to datasets from multiple labs, facilitating replication and data re-use. The system can be extended by adding new pipelines to the existing database as better tools and algorithms become available. These features enable the re-analysis of data to examine how the results depend on the choice of algorithm. We believe it is critical to provide this kind of future-compatibility to maximize the impact of the years of experimental work that go into each dataset.

Limitations

Although Spyglass provides many useful features for reproducible data analysis, it has several limitations. For example, because of the central role played by the NWB format in Spyglass, a potential user must first convert their data to NWB, which requires time and effort⁴². In addition, users are expected to set up and maintain a relational database, which may require additional training. This includes learning to work with the structure of DataJoint, such as the strict data integrity requirement that makes modification of existing tables difficult. Finally, Spyglass does not yet include pipelines for processing certain types of neural data, such as optical physiology or fiber photometry.

Fortunately, there are ongoing efforts to address these challenges. These include tools to simplify the raw data conversion into NWB, such as NeuroConv, a package to convert neurophysiology data in common formats to NWB automatically, and NWB GUIDE, a desktop app that guides users through the process of converting data to NWB without writing any code. Furthermore, we provide many tutorials on the documentation website so that the user can efficiently learn to use Spyglass. We continue to actively maintain Spyglass and are eager to work with the community to extend it and support data types and analyses beyond what is currently available. These efforts will increase the usability and reach of Spyglass and make its adoption more attractive, particularly to early-stage investigators.

Future applications

Spyglass and similar tools have the potential to transform scientific data analysis. In addition to facilitating examination or extension of published results, they enable meta-analysis across studies and easy testing of novel methods across multiple datasets. The machine-readable form of data and analysis pipelines also opens doors for machine-driven analysis and hypothesis testing. As these tools develop and become more accessible, we believe that frameworks like Spyglass will likely become essential for neuroscience researchers.

Methods and materials

Coding environment

Spyglass was developed in Python 3.9 and is compatible with version 3.10 as well. See our dependency list for a full list of Python packages used.

NWB conversion

To facilitate conversion of raw data to NWB format, we offer trodes-to-NWB, a sister package to Spyglass for converting data acquired with the SpikeGadgets hardware to NWB. This comes with a web-based GUI for conveniently generating a YAML file containing the metadata used by trodes-to-NWB. For converting data not acquired with SpikeGadgets, users can use NWB conversion tools developed by the NWB team, such as NeuroConv and NWB GUIDE.

NWB file conventions

We adopted a specific set of conventions for our NWB files. Some of these conventions rely on a specific set of Frank lab-specific NWB extensions:

• Time:

  • Spyglass inherits from the source NWB file either the explicit or implicit timestamps. NWB files from Frank lab have explicit timestamps for each sample in Unix time (seconds since 12:00 am January 1^st, 1970). This lets users to know exactly when data were collected. Spyglass is also compatible with other approaches, however, including implicit timestamping consisting of the start time and sampling rate.

• ElectrodeTable and ElectrodeGroup:

  • ElectrodeGroups are stored in a custom NWB extension that also includes the name of the targeted brain region for each group.

  • The NWB file contains information about the relative locations of each of the electrodes within each physical device used for data collection. This ensures that the relative locations of the electrodes are available for spike sorting and registration to histology.

• Video files

  • The relative path to the video files collected along with the recordings are stored in the NWB file.

• Additional files

  • Other files important to recreate the conditions of the experiments can be saved, depending on the format. For example, the code used for implementing the behavioral paradigm or reward contingency can be stored as text objects in the NWB file.

NWB file ingestion

Although the NWB format serves as a community standard for neurophysiology data and has a list of best practices, it allows some flexibility in the specification of data within NWB files to accommodate user preferences. For example, the ElectricalSeries object that stores the electrophysiology data may have different names depending on the convention chosen by the investigator, which may complicate programmatic access to the data. To make Spyglass interoperable with NWB files of varying degrees of NWB-compliance, we have created an option to supply or override information that is missing in the NWB file but is nevertheless required by Spyglass via a configuration file that can accompany the NWB file. We provide an example of this approach in a tutorial (Table 1, 02_Insert_Data).

Permission-handling and cautious delete

Spyglass is based on a relational database that is accessible to multiple users. In some cases, the type of operations that can be applied to individual data entries (i.e., rows of a table) may need to be restricted to a specified set of users. This is particularly true for operations that are irreversible or time consuming, such as deleting a row from a table storing analysis results. However, there is no inherent mechanism within MySQL or DataJoint that allows permission handling at the level of individual rows of a table. To solve this problem, we have implemented a cautious_delete function, in which the user’s permission to carry out a delete operation is checked before it is applied. The permission is granted based on team membership within the lab, reflected in the LabTeam table. Though this is not a formal permission-management system, it can prevent accidental deletions.

Sharing files via Kachery

One way to share the results of Spyglass analysis pipelines is to make the database publicly available. This gives anyone the permission to access the rows of the tables that make up the pipelines and inspect the metadata and the parameters associated with each step of the analysis. But because Spyglass only saves a path to the NWB files containing analysis results within the tables, external viewers cannot download the data and examine it by default.

To enable controlled external access to the data, we have created a system to share selected analysis NWB files with a specified group of users via Kachery. We define a set of tables (KacheryZone and AnalysisNWBfileKachery) where users can associate analysis NWB files to be shared with a Kachery Zone, making it available to all remote clients who are members of the zone through cloud storage services like Cloudflare R2 bucket or self-hosted servers. Once linked, Spyglass automatically requests, downloads, and manages analysis data for remote users attempting to access shared data through Spyglass tables. This provides a convenient way to provide access to the Spyglass pipelines and associated data files to collaborators.

Customizing pipelines

To alleviate the challenges associated with database design, we have identified design principles that have been tested extensively by multiple users in the Frank lab. These are described in the text and illustrated with examples in Figures 2 and 3. We recommend users adopt these design elements for building their custom pipelines. We also describe the naming conventions for the tables defined as Python classes and important methods associated with them (e.g. for multiple versions of a pipeline) in our Developer Notes available online. Once the pipeline is sufficiently mature and potentially useful to other scientists, we encourage users to submit their pipelines as a pull request to our GitHub repository.

Decoding of position from NWB files from multiple laboratories

The Frank lab data is available on the DANDI archive (DANDI:000937). The Buzsáki lab data was also obtained from DANDI (DANDI:000059/0.230907.2101). For decoding the Frank lab data, we applied the clusterless decoding pipeline by detecting the amplitude of threshold-crossing events in the tetrode recordings. For decoding the Buzsáki lab data, we applied a sorted-spikes decoding pipeline. The code for these decoding pipelines, as well as detailed tutorials describing them, are available online (Table 1, 40_Extracting_Clusterless_Waveform_Features, 41_Decoding_Clusterless, 42_Decoding_SortedSpikes). Code to generate Figure 5 can be found at: https://github.com/LorenFrankLab/spyglass-paper. Briefly, decoding the latent neural position and extracting the distance between the most likely decoded position and the animal’s position used methods described in Denovellis et al. (2021). We used a timestep of 4 ms and a position bin size of 2 cm with a continuous (6 cm variance Gaussian random walk) and fragmented

(uniform distribution) discrete state. Place intensity receptive fields were estimated using a Gaussian kernel density estimate with a standard deviation of 6 cm for position and 24 mV for amplitude space (amplitude space was used for the clusterless analysis only). We calculated the power of the multiunit firing rate and the decoded distance from the animal by using a multitaper spectrogram during the pre-cooling and cooling periods. The time resolution was 3 seconds and the frequency resolution of 2/3 Hz with a single taper. We excluded immobility periods by using a threshold of 10 cm/s. Power difference was calculated by converting to the Decibel scale and taking the difference of average power under the cooling and pre-cooling condition. The decoded speed of theta sequences was calculated by taking the absolute value of the second-order difference of the decoded distance from the animal (function numpy.gradient) multiplied by the sampling frequency (250 Hz).

Acknowledgements

We thank members of the Frank and Gillespie laboratory for bug reports and testing. We also thank Peter Petersen for consultations about analyzing his publicly available data, Daniel Liu for initial discussions about standardization of pipelines, and Abhilasha Joshi for consulting on the DeepLabCut pipeline. Finally, we thank Vanessa Bender for comments on the manuscript. This work was supported by HHMI funds and NIH grants RF1MH130623 and RF1MH133778 to L.M.F. and a Helen Hay Whitney Postdoctoral Fellowship to K.H.L.

Additional information

Author contributions

Funding

National Institutes of Health (RF1MH130623)

National Institutes of Health (RF1MH133778)

Helen Hay Whitney Foundation

Howard Hughes Medical Institute