High frequency spike inference with particle Gibbs sampling

Reviewer #1 (Public Review):

Summary:

In this study, Diana et al. present a Monte Carlo-based method to perform spike inference from calcium imaging data. A particular strength of their approach is that they can estimate not only averages but also uncertainties of the modeled process. The authors then focus on the quantification of spike time uncertainties in simulated data and in data recorded with a high sampling rate in cerebellar slices with GCaMP8f.

Strengths:

- The authors provide a solid groundwork for sequential Monte Carlo-based spike inference, which extends previous work of Pnevmatikakis et al., Greenberg et al., and others.

- The integration of two states (silence vs. burst firing) seems to improve the performance of the model.

- The acquisition of a GCaMP8f dataset in the cerebellum is useful and helps make the point that high spike time inference precision is possible under certain conditions.

Weaknesses:

- The algorithm is designed to predict single spike times. Currently, it is not benchmarked against other algorithms in terms of single spike precision and spike time errors. A benchmarking with the most recent other SMC model and another good model focused on single spike outputs (e.g., MLSpike) would be useful to have.

- Some of the analyses and benchmarks seem too cursory, and the reporting simply consists of a visual impression of results instead of proper analysis and quantification. For example, the authors write "The spike patterns obtained using our method are very similar across trials, showing that PGBAR can reliably detect single-trial action potential-evoked GCaMP8f fluorescence transients." This is a highly qualitative statement, just based on the (subjective) visual impression of a plot. Similarly, the authors write "we could reliably identify the two spikes in each trial", but this claim is not supported by quantification or a figure, as far as I can see. The authors write "but the trade-off between temporal accuracy, SNR and sampling frequency must be considered", but they don't discuss these trade-offs systematically.

- It has been shown several times from experimental data that spike inference with single spike resolution does not work well (Huang et al. eLife, 2021; Rupprecht et al., Nature Neuroscience, 2021) in general. This limitation should be discussed with respect to the applicability of the proposed algorithm for standard population calcium imaging data.

- Several analyses are based on artificial, simulated data with simplifying assumptions. Ever since Theis et al., Neuron, 2016, it has been known that artificially generated ground truth data should not be used as the primary means to evaluate spike inference algorithms. It would have been informative if the authors had used either the CASCADE dataset or their cerebellum dataset for more detailed analyses, in particular of single spike time precision.

- In its current state, the sum of the current weaknesses makes the suggested method, while interesting for experts, rather unattractive for experimentalists who want to perform spike inference on their recorded calcium imaging data.

Other comments:

- One of the key features of the SMC model is the assumption of two states (bursting vs. non-bursting). However, while it seems clear that this approach is helpful, it is not clear where this idea comes from, from an observation of the data or another concept.

- Another SMC algorithm (Greenberg et al., 2018) stated that the fitted parameters showed some degeneracy, resulting in ambiguous fitting parameters. It would be good to know if this problem was avoided by the authors.

Reviewer #2 (Public Review):

Summary:

Methods to infer action potentials from fluorescence-based measurements of intracellular calcium dynamics are important for optical measurements of activity across large populations of neurons. The variety of existing methods can be separated into two broad classes: a) model-independent approaches that are trained on ground truth datasets (e.g., deep networks), and b) approaches based on a model of the processes that link action potentials to calcium signals. Models usually contain parameters describing biophysical variables, such as rate constants of the calcium dynamics and features of the calcium indicator. The method presented here, PGBAR, is model-based and uses a Bayesian approach. A novelty of PGBAR is that static parameters and state variables are jointly estimated using particle Gibbs sampling, a sequential Monte Carlo technique that can efficiently sample the latent embedding space.

Strengths:

A main strength of PGBAR is that it provides probability distributions rather than point estimates of spike times. This is different from most other methods and may be an important feature in cases when estimates of uncertainty are desired. Another important feature of PGBAR is that it estimates not only the state variable representing spiking activity but also other variables such as baseline fluctuations and stationary model variables, in a joint process. PGBAR can therefore provide more information than various other methods. The information in the GitHub repository is well-organized.

Weaknesses:

On the other hand, the accuracy of spike train reconstructions is not higher than that of other model-based approaches, and clearly lower than the accuracy of a model-independent approach based on a deep network. The authors demonstrate convincingly that PGBAR can resolve inter-spike intervals in the range of 5 ms using fluorescence data obtained with a very fast genetically encoded calcium indicator at very high sampling rates (line scans at >= 1 kHz). It would be interesting to more systematically compare the performance of PGBAR to other methods in this regime of high temporal resolution, which has not been explored much.