The hippocampus plays a central role in the encoding and consolidation of new memories (Eichenbaum, 2000; Frankland and Bontempi, 2005; Klinzing et al., 2019; Lewis and Bendor, 2019; Squire, 1992). During locomotion in rodents, hippocampal place cells are active in specific regions of the animal’s environment (place fields), resulting in a sequential pattern of place cell firing when the animal runs along a spatial trajectory (O’Keefe and Dostrovsky, 1971). During offline states, such as quiet restfulness and non-REM sleep, the sequential pattern of neural activity observed during behavior is spontaneously reactivated, a phenomenon referred to as ‘hippocampal replay’ (Lee and Wilson, 2002; Wilson and McNaughton, 1994). Offline replay of a neural memory trace is postulated to be a central mechanism by which memories recently encoded in the hippocampus can be consolidated by further stabilizing these memories in distributed cortico-hippocampal circuits for long-term storage (Buzsáki, 1989; Klinzing et al., 2019).

Evidence of replay was first discovered almost 30 years ago with the demonstration that during sleep, temporal correlations can be observed between the co-firing of place cell pairs with overlapping place fields (Wilson and McNaughton, 1994). Since then, methods for large-scale chronic extracellular neural recordings have become more advanced, allowing the simultaneous recording of tens and even hundreds of neurons (Ji and Wilson, 2007; Lee and Wilson, 2002; Pfeiffer and Foster, 2013; Widloski and Foster, 2022). In line with these developments, the analysis methods for replay have also become more sophisticated - shifting from the pairwise analysis of place cells to detecting sequential patterns within neuronal ensembles, using either spiking activity directly (Foster and Wilson, 2006) or decoding this activity to extrapolate the virtual spatial trajectories replaying (Davidson et al., 2009; Zhang et al., 1998). Commonly used replay scoring metrics for quantifying the fidelity of a replay sequence include: (1) a Spearman’s rank-order correlation of spike times, which quantifies the ordinal relationship between the temporal order of place cell firing during behavior and a replay event’s spike train (Foster and Wilson, 2006), but assumes that the place cell sequence is ordered accordingly to each place cell’s peak firing rate location alone, (2) a weighted correlation of the decoded replay event, which quantifies a generalized linear correlation in time and position weighted by the decoded posterior probabilities without any assumption about the temporal rigidity of the replayed trajectory (Grosmark and Buzsáki, 2016; Silva et al., 2015; Tirole et al., 2022), and (3) a linear fitting of the decoded replay event, which finds the linear path with the maximum summed decoded probability, assuming that the trajectory’s slope is constant (Davidson et al., 2009; Gomperts et al., 2015; Ólafsdóttir et al., 2017). However, because replay is generated by an internal and spontaneous state of the brain, there is no external reference to indicate whether a given replay event is truly a reinstatement of a memory trace. Without a ground truth, the detection of replay events must be inferred based on whether the statistical likelihood of observing a match between the sequential structure of the replayed event and the original behavioral template is sufficiently low, by chance. To quantify this, each event’s replay score is compared to a distribution of scores obtained using randomized data, permutated in either the spatial or temporal domain (i.e. a shuffled distribution), where statistically significant replay scores must be greater than a certain percentage of this shuffled distribution, typically 95% for an alpha level (α)<0.05.

While recent replay studies have relied predominately on these three major methods of scoring replay (rank-order, weighted correlation, or linear fit), there are still many variations in how these scores can be calculated and how the subsequent statistical significance of these scores are measured. This can lead to issues in reproducibility and a greater difficulty in interpreting conflicting results between studies (Tingley and Peyrache, 2020). To overcome this problem, we need the ability to cross-check replay events (i.e. are the real events being correctly detected, and non-events being correctly rejected), in spite of not having a ground truth (Tingley and Peyrache, 2020; van der Meer et al., 2017; van der Meer et al., 2020). Hypothetically, any method of cross-checking should at the very least be able to pass a basic test of distinguishing between replay events detected from real data and spurious replay events detected from randomized data.

Given that we are detecting a replay sequence using a replay score (and its significance level α), we cannot use similar metrics for cross-checking due to a lack of independence. Here, we solve this problem by evaluating replay sequence events using a sequenceless decoding approach, which has been used in several recent replay studies when the rat’s behavior involved running different trajectories, either on a T-maze with two arms or on two different linear tracks (Carey et al., 2019; Tirole et al., 2022). This framework quantifies how well the sequence-based replay detector discriminates between two track-specific sequences, with a greater difference in summed trajectory likelihoods indicating a higher discriminability.

Several underlying assumptions are required for this framework: (1) for a given replay detection method, the proportion of spurious events (generated from randomized data) that are detected as significant events can be used to empirically estimate the proportion of non-replay events that are falsely labeled as significant replay events in real hippocampal data. However, this assumes that the null distribution used for creating randomized data is not used for detection, and that it does not underestimate the false-positive rates compared to the other null distribution(s) used for detection, (2) for neural data, pooled from multiple animals, with track-specific replay sequences but not spurious replay events, track discriminability should correlate with the sequenceness score of the track-specific replay, and (3) the empirically estimated false-positive rate can vary between methods, but can be adjusted by a scaling correction of the alpha level such that the performance of different replay methods can be evaluated and compared at an equivalent empirically estimated false-positive rate. Note that this method of scaling the alpha level is designed for method comparison only and should not be meant as a substitute for using appropriate shuffle methods to create a sufficient set of null distributions for detection.

Based on replay data pooled from five rats running on two novel tracks, we validate this framework and demonstrate how measures of sequence fidelity combined with measures of sequenceless track discriminability provide the means to evaluate and compare the performance of different sequence-based replay detection strategies and methods.

Extracellular signals from the dorsal CA1 region of the hippocampus were recorded in rats running back and forth on two novel linear tracks (male Lister-hooded, n=5, 10 sessions, dataset from Tirole et al., 2022). In addition to running on two linear tracks (RUN), rats also had a rest/sleep session in a remote location both before (PRE) and afterward (POST) (Figure 1A). To demonstrate how this framework quantifies replay detection performance in terms of dataset-specific detection rate, false-positive rate, and track discriminability, we started by detecting replay events using the weighted correlation of the decoded event, a common sequence-based replay detection approach (Grosmark and Buzsáki, 2016).

Figure 1 with 4 supplements see all

Download asset Open asset

Figure 1—source data 1

Summary of replay detection performance for weighted correlation method with two shuffles.

https://cdn.elifesciences.org/articles/85635/elife-85635-fig1-data1-v2.xlsx

Download elife-85635-fig1-data1-v2.xlsx

Candidate replay events were population burst events (peak z-scored multi-unit activity [MUA]>3) that occurred during periods of inactivity, when the animal’s velocity is less than 5 cm/s (see Methods). Across 10 sessions, we detected 8485, 4643, and 14,326 candidate events in PRE, RUN, and POST session, respectively. Detected replay events were required to also have a ripple power z-score greater than 3 and a significant replay score - i.e., higher than 95% of the scores obtained from each of two shuffle distributions, namely a place field circular shuffle and a time bin permutation shuffle (see Methods) (Grosmark and Buzsáki, 2016). The replay score was obtained by performing a weighted correlation on the posterior probabilities (across place and time) obtained using a naïve Bayesian decoder (Figure 1B and C). Because the dataset includes more than one track, the posterior probabilities across the two available tracks were normalized at each time bin such that their combined sum was one (Figure 1B; Bendor and Wilson, 2012; Carey et al., 2019).

Ripple power is an important criterion for replay event selection during POST

During both quiet wakefulness and non-REM sleep, replay events preferentially occur during sharp-wave ripple events, transient high-frequency oscillations (150–250 Hz) in the local field potential (LFP) recorded near the cell layer of CA1 (Buzsáki, 2015; Foster and Wilson, 2006; Lee and Wilson, 2002). As such, a minimum ripple power has been used as a criterion for detecting candidate replay events prior to sequence detection, however the threshold has varied substantially across previous studies ranging from 2 SD (standard deviations) above baseline (Diba and Buzsáki, 2007; Gillespie et al., 2021) to 7 SD above baseline (Csicsvari et al., 2007; Ji and Wilson, 2007). However, because many replay events consist of the spontaneous reactivation of a large proportion of place cells within a short time window (more than typically occurs during active behavior), a substantial increase in MUA during immobility also provides a reasonable criterion for detecting candidate replay events (Davidson et al., 2009). Because using only MUA as a threshold typically leads to more candidate replay events, and is believed to be more reliable for detecting both the start and end times of the replay event, many recent studies have opted to no longer use minimum ripple power as a criterion for candidate replay events (Gomperts et al., 2015; Ólafsdóttir et al., 2018; Ólafsdóttir et al., 2016; Silva et al., 2015).

We next applied our replay analysis framework to test how a stricter criterion for ripple power affects track discriminability. Candidate events were selected based on both elevated MUA (z-score>3) and ripple power limited to a specific range, measured in SD above baseline (i.e. a z-score of 0–3, 3–5, 5–10, or >10). For equivalent alpha levels, a stricter ripple threshold resulted in both a higher proportion of significant events and a higher mean log odds difference (Figure 2A and B). The increase in track discriminability observed with higher ripple power was not associated with a change in the number of active place cells or total place cell spiking activity (Figure 2—figure supplement 1). However, one major difference between RUN and POST replay events was that even at the lowest ripple power threshold, RUN replay events had a non-zero mean log odds difference, which increased with stricter alpha levels. In contrast for POST replay events with a ripple power less than a z-score of 5, the mean log odds difference was near zero for all alpha levels tested. Furthermore, while the proportion of detected POST events with a ripple power less than a z-score of 5 was approximately 20% at an alpha level of 0.05, the proportion of detected POST events dropped to near chance levels of 10% when the alpha level was adjusted to match the mean false-positive rate of 5% (where 10% of spurious events were detected as significant events across both tracks) (Figure 2C and D and Figure 2—figure supplement 2, see Methods). For ripple thresholds above 5, the log odds difference increased as the proportion of events and the alpha level decreased (Figure 2A and B). Similar results were observed if stricter criteria were used in detecting candidate replay events - either lowering the speed threshold criterion (<1 cm/s) or imposing an additional requirement for low theta power (z-score<0) (Figure 2—figure supplement 3), suggesting that our finding was not confounded by the presence of other neuronal sequences during movement such as theta sequences. Next, we tested how shifting the ripple threshold (between 0 and 10 SD above baseline) could impact overall detection rates (Figure 2—figure supplement 4), calculating the proportion of significant events out of all of the candidate events based on MUA criteria alone. Consistent with our main finding, while the proportion of significant events dropped as a result of ripple thresholding, only replay during POST but not RUN improved its log odds difference with a stricter ripple threshold. These results emphasize that a ripple power threshold is not necessary for RUN replay events in our dataset but may still be beneficial, as long as it does not reject too many good replay events with low ripple power. In other words, depending on the experimental design, it is possible that a stricter alpha level with no ripple threshold can be used to detect more replay events than using a less strict alpha level combined with a strict ripple power threshold. However, for POST replay events, a threshold at least in the range of a z-score of 3–5 is recommended based on our dataset, to reduce inclusion of potential false-positives within the pool of detected replay events. For the remainder of this study, we incorporated a strict, albeit less conservative ripple power threshold (z-score>3) for both RUN and POST, given the trade-off between an improved track discriminability and a decrease in the number of detected events. This also allowed a more direct comparison of awake and rest replay using identical criteria while still being consistent with previous replay studies (Karlsson and Frank, 2009; Pfeiffer and Foster, 2013; Todorova and Zugaro, 2019).

Figure 2 with 4 supplements see all

Download asset Open asset

Figure 2—source data 1

Summary of replay detection performance at different ripple power thresholds.

https://cdn.elifesciences.org/articles/85635/elife-85635-fig2-data1-v2.xlsx

Download elife-85635-fig2-data1-v2.xlsx

Replay detection is sensitive to the shuffle method used

Statistical significance of each replay event is calculated by comparing the replay score to one or more Monte-Carlo shuffle distributions, with each shuffle designed to randomize a specific aspect of sequential place cell firing while attempting to keep other factors intact. Some shuffles are applied to randomize the place fields or the spike train of the original data prior to Bayesian decoding, while other shuffles are applied to the spatial or temporal dimension of the decoded posterior probabilities. Because place cells often fire bursts of spikes, this creates non-independent samples, which violate the assumption of having independent samples for a statistical test. Thus, it is important for a shuffle method to preserve such aspects of the data that are not independent, to avoid adding type 1 errors in the statistical analysis, leading to a false-positive rate that exceeds the alpha level. Given this, we next examined how different shuffling procedures impact replay detection performance. Using the weighted correlation scoring method, we examined four types of shuffling procedures to see if they differed in how they detected replay during RUN and POST epochs. These consisted of two pre-decoding shuffles (spike train circular shuffle, place field circular shuffle) and two post-decoding shuffles (place bin circular shuffle, time bin permutation shuffle) (Figure 3A–D, Figure 3—figure supplement 1).

Figure 3 with 1 supplement see all

Download asset Open asset

Figure 3—source data 1

Summary of replay detection performance using four different shuflling methods.

https://cdn.elifesciences.org/articles/85635/elife-85635-fig3-data1-v2.xlsx

Download elife-85635-fig3-data1-v2.xlsx

For both RUN and POST, we observed that pre-decoding shuffles performed better than post-decoding shuffles, with the time bin permutation shuffle showing a significantly lower log odd difference compared to the other three shuffles (Figure 3A and B). Although the post-decoding time bin permutation shuffle seemingly led to the highest detection rate at an alpha level of 0.05 (Figure 3C), we observed that its mean false-positive rate at an alpha level of 0.05 was also substantially higher than other shuffling procedures. When we adjusted the alpha level to match a mean false-positive rate of 5%, the post-decoding time bin permutation shuffle had the poorest performance (the lowest proportion of significant events and the lowest mean log odds difference) across the four types of shuffles for both RUN and POST epochs (Figure 3D). For both the RUN and POST epochs, the remaining three shuffling procedures had a similar mean log odds difference when using an FPR-matched alpha level. However, while the proportion of significant events were similar across the three shuffling methods for POST, the two pre-decoding shuffling procedures had a higher proportion of significant events for RUN replay events. These results suggested that the shuffling procedures that directly manipulated the place field data or spike train data before decoding may be more efficacious than procedures that randomized the posterior probability distributions after decoding.

Replay detection can be improved by adding stricter detection criteria

Following our comparison of different shuffling procedures, we next investigated how the inclusion of additional criteria in replay detection could further improve performance. Commonly this is accomplished by using multiple types of shuffling procedures, with the requirement that each shuffling method must independently pass the same alpha level for the replay event to be classified as statistically significant. An alternative approach to using multiple shuffling procedures is to impose a criteria of jump distance - the number of spatial bins the decoded estimated position is allowed to jump across neighboring time bins during a spatial trajectory, with candidate replay events rejected if they fail this criterion (Silva et al., 2015). One justification for using this metric is that a lower jump distance is correlated with a higher weighted correlation score, however not all events with higher scores are necessarily higher quality events.

Based on this, we compared the performance of four replay detection methods varying in their shuffling procedures: (1) a single place field circular shuffle, (2) a place field circular shuffle with a jump distance threshold (40% of track) (Silva et al., 2015), (3) two shuffles (place field circular shuffle and time bin permutation shuffle) (Grosmark and Buzsáki, 2016), and (4) three shuffles (place field circular shuffle, spike train circular shuffle, and place bin circular shuffle) (Tirole et al., 2022). For both RUN and POST epochs, increasing the number of shuffles increased the mean log odd difference and decreased the proportion of significant events and the estimated false-positive rate, for similar alpha levels (Figure 4A and B and Figure 4—figure supplement 1A and B). Adding a jump distance threshold as an additional criterion also increased the mean log odd difference, although this was accompanied by a marked reduction in the proportion of significant replay events detected (Figure 4A and B). Before controlling for the estimated false-positive rates associated with each method, as expected, the stricter methods would detect fewer replay events than more permissive methods. For instance, at alpha level of 0.05, around 50% of candidate events would be detected as significant events when using single shuffle, as opposed to 40% when using two shuffles (Figure 4C). However, after adjusting the alpha level to a matching false-positive rate of 0.05, we found that stricter methods would detect more replay events. For example, at an FPR-matched alpha level, the proportion of significant events would reduce to 32% for a single shuffle (α = 0.007) but 35% for the stricter two-shuffle method (α = 0.032). This shows that controlling for the estimated false-positive rate can, almost paradoxically, result in stricter methods with additional shuffle types detecting more replay events. When using the original alpha level of 0.05, two methods (1 shuffle with a jump distance threshold and 3 shuffles) resulted in the highest mean log odd difference (Figure 4C). However, if an FPR-matched alpha level was used (with the estimated false-positive rate fixed at 0.05), only the use of 2 or more shuffles achieved both the highest proportion of events and largest mean log odds difference, for both RUN and POST epochs (Figure 4D). In contrast to this, a single shuffle with an added jump distance criterion performed the poorest (less events and lower track discriminability) out of the four methods when using the FPR-matched alpha level. This suggests that it is more beneficial to use a stricter alpha level than to impose a jump distance, an observation that extended to a range of different jump distances spanning 20–60% of the track (Figure 4—figure supplement 2).

Figure 4 with 2 supplements see all

Download asset Open asset

Figure 4—source data 1

Summary of replay detection performance when adding stricter detection criteria.

https://cdn.elifesciences.org/articles/85635/elife-85635-fig4-data1-v2.xlsx

Download elife-85635-fig4-data1-v2.xlsx

Rank-order-based replay detection is sensitive to spike selection

We next extended our framework to investigate other replay scoring methods - first focusing on the use of rank-order correlation. Instead of scoring the decoded trajectories, this approach directly quantifies the ordinal relationship within the spiking patterns (Foster and Wilson, 2006). In this method, the sequential order of place fields is ranked along the track, and compared with the corresponding rank of the spike times within the replay event. Rank-order correlation has been previously applied in several ways: (1) to all spikes within the replay event (Dragoi and Tonegawa, 2011), (2) to all proto-events, each representing a bursts of spikes (Foster and Wilson, 2006), or (3) only to one spike emitted by each place cell during a candidate replay event (e.g. median or first spike) (Tirole et al., 2022). However, as mentioned previously, place cells often fire bursts of spikes which in effect creates non-independent samples. Therefore, in theory, the inclusion of all spikes for the Spearman’s rank-order based analysis would violate the statistical assumption of independent samples and lead to more type I errors (and a higher false-positive rate). We compared the rank-order-based analysis using either all spikes or median spike in our framework, and found that the estimated false-positive rate when using independent events (median spike time) remained around 5% (lower CI = 5.2% and upper CI = 5.8%), but increased to around 17% (lower CI = 16.3% and upper CI = 17.2%), when using all spikes (Figure 5 and Figure 5—figure supplement 1). However, when using the FPR-matched alpha level (false-positive rate fixed at 5%), the trade-off between the proportion of significant events and their mean log odds difference was similar between the two methods (Figure 5D and Figure 5—figure supplement 1).

Figure 5 with 1 supplement see all

Download asset Open asset

Figure 5—source data 1

Summary of replay detection performance for rank-order-based methods.

https://cdn.elifesciences.org/articles/85635/elife-85635-fig5-data1-v2.xlsx

Download elife-85635-fig5-data1-v2.xlsx

Replay events have a similar quality across different detection methods, when using an FPR-matched alpha level, but vary in their proportion

Finally, we expanded our analysis to directly compare every replay scoring and shuffling method already implemented in this report, with the addition one more commonly used method - a linear fit-based score of decoded trajectories with either one and two types of shuffles (Davidson et al., 2009; Muessig et al., 2019; Ólafsdóttir et al., 2017). In the linear fit approach, the decoded posterior probability matrix, representing the probability of the virtual trajectory at each position on the track and at each time point, is compared to the best linear trajectory; the corresponding replay score is the sum of all posterior probabilities within a certain distance of this line, after maximizing this score across all possible lines that can pass through this matrix. While the linear fit approach is more stringent than the weighted correlation, it may also be less sensitive to non-linear replay trajectories (e.g. sigmoidal). When using a cell-id permuted randomized dataset to examine the chance-level detection for each method, we observed no bias in the mean log odds difference for every method, except for a linear fit approach with two shuffles (Figure 6—figure supplement 1). To avoid any possible intrinsic bias in our calculation of the mean log odds, we calculated the adjusted mean log odds difference, by subtracting any potential bias in log odds difference measured using spurious replay events (Figure 6). Similar results were observed using an alternative within-track normalization approach in Bayesian decoding prior to replay detection (Figure 6—figure supplement 2). Finally, to provide a more comprehensive comparison across methods, we also included replay events detected in the PRE epochs, a phenomenon referred to as de novo preplay where statistically significant sequences are detected prior to any knowledge or experience running on the track (Dragoi and Tonegawa, 2011).

Figure 6 with 6 supplements see all

Download asset Open asset

Figure 6—source data 1

Summary of replay detection performance across multiple detection methods.

https://cdn.elifesciences.org/articles/85635/elife-85635-fig6-data1-v2.xlsx

Download elife-85635-fig6-data1-v2.xlsx

Using our framework to compare replay detection methodologies, which incorporated (1) the corrected mean log odds difference, (2) the empirically estimated mean false-positive rate based on replay detection using a randomized dataset (cell-id permutation), and (3) the proportion of significant events on both tracks, we observed four main findings regarding replay during three different behavioral epochs (Figure 6 and Figure 6—figure supplements 3–5). First, the proportion of significant events varied substantially across methods when using the original alpha level of 0.05, however this was consistent with the estimated false-positive rate, which ranged from 2% to 18%. Moreover, the proportion of mean false-positive rates estimated were also highly correlated with the proportion of events being classified as significant for both tracks (dual-track events), suggesting that most dual-track events we observed were likely due to false-positive detection on one of the two tracks rather than multiple track representations in a single replay event (Figure 6—figure supplement 6). Second, using the FPR-matched alpha level, all methods had a proportion of significant preplay events close to 10%, which was similar to the proportion of false-positive events expected by chance for two tracks. Furthermore, the shuffle-subtracted mean log odds difference for preplay was approximately zero for all methods, suggesting that regardless of the method used, if used correctly, preplay is indistinguishable from randomly generated spurious replay events. Third, using the FPR-matched alpha level for POST replay events, all methods produced a corrected mean log odds difference near 1, however linear and weighted correlation methods with 2 or more shuffles detected a higher proportion of significant events. Finally, using the FPR-matched alpha level for RUN replay events, the majority of methods produced a corrected mean log odds difference near 2. Despite using the FPR-matched alpha level to achieve the same mean false-positive rate, we observed that using two or more shuffling procedures would lead to significant improvement in the overall replay detection performance for both weighted correlation and linear fitting method. Replay detection methods will continue to evolve and be refined in the future; the quantification of any improvements is now possible with the framework outlined here.

Previous studies describing the phenomenon of replay have used a wide variety of methods and criteria for replay detection. The most common methods in the last decade have been a rank-order correlation (applied to spike trains), and the scoring of decoded trajectories (using a weighted correlation or linear fit) (Foster, 2017; Tingley and Peyrache, 2020; van der Meer et al., 2020). However, even within these two classes of methods, there is substantial variation in the parameters and strategies for statistical testing, including the type and number of shuffling procedures, and the use of ripple power or jump distance as a threshold. The diversity in replay methods prevents an unbiased comparison between different replay findings. In particular, while replay detection statistical tests are commonly used with an alpha level of 0.05, this value does not necessarily match the actual false-positive detection rate for all methods. This issue is not just limited to when the entire spike train is used to calculate the rank-order correlation (as discussed above), but also applies to all methods using decoded neural activity. The use of either spatially smoothed place fields or spike bursts spanning multiple time bins similarly violates the assumption of independent samples. While the shuffling of neural patterns comprising the replayed trajectory may compensate for non-independent samples, its effectiveness varies substantially, especially given the wide range of parameters used across the published replay literature. Furthermore, while imposing stricter criteria for replay detection would theoretically improve the average fidelity of the events and reduce the false-positive rate, it comes with the caveat of potentially increasing the rejection of true replay events.

Quantifying and comparing replay detection performance has been an ongoing challenge due to the lack of a ground truth. Taking advantage of data collected from rats running on two linear tracks, here we introduce a novel framework for comparing different sequence-based replay detection methods - quantifying how well a given replay detection method correctly discriminates track-specific replay events. In particular, this framework quantifies the proportion of significant replay events (quantity) and the mean log odds difference (track discriminability) of a collection of track-specific replay events detected as well as adjusting the alpha level used for replay detection according to an estimated false-positive rate of 5% based on cell-id randomized dataset. Using this approach, we make six key observations: (1) For candidate events selected using an MUA threshold, the additional step of including a ripple power threshold is not necessary for RUN replay but is recommended for POST replay events. For our dataset, the POST replay events with ripple power below a z-score of 3–5 were indistinguishable from spurious events. While the exact ripple z-score threshold to implement may differ depending on the experimental condition (e.g. electrode placement, behavioral paradigm, noise level, etc.) and experimental aim, our findings highlight the benefit of using a ripple power threshold for detecting replay during POST. (2) More shuffles, with a preference for pre-decoding shuffles, lead to better replay detection, even after the alpha level is adjusted to match an estimated false-positive rate. (3) Without adjusting the alpha level, there is a very wide distribution of both the track discriminability and the proportion of significant events detected across methods. False-positive rates also vary substantially, and one should not assume that replay exists simply because the proportion of significant events detected is greater than the alpha level (typically 5%) used in detection. (4) After adjusting the alpha level to match an estimated false-positive rate, there is greater similarity observed between methods, however using either a weighted correlation or a linear fit replay scoring with only one shuffling procedure still results in the worst overall performance for our dataset. (5) Our metric for replay track discriminability yields a similar result across many methods once the alpha level is adjusted to fix the estimated false-positive rate at 5%, however the proportion of replay events detected remains more variable. (6) Regardless of the method used after the alpha level is adjusted, preplay events are at chance level for both track discriminability (mean log odds difference of 0) and proportion detected (10% of events across two tracks). Previous studies have used an alpha level of 0.05, which in our hands substantially changes the proportion of replay events detected across methods - overly strict methods such as weighted correlation with a jump distance criterion would have a proportion less than 5%, while more lenient methods such as rank-order correlation with all spikes included can have a proportion closer to 30%. Therefore, without actually measuring a false-positive rate, one cannot assume that a higher proportion of events relative to the alpha level is evidence, on its own, for the existence of replay. In all methods tested, we do not see any evidence of preplay events behaving differently from spurious replay events generated using randomized data. In contrast to this, both POST and RUN replay events have a proportion of significant events substantially above the chance level for detection, for any method used with an FPR-matched alpha level.

Here, we have focused on a number of common replay scoring methods and criteria, however there are many variations and nuanced differences between published methods, that were beyond the scope of this study to test. For instance, we did not examine the impact of spatial smoothing (place fields) and temporal smoothing (posterior probability matrix and/or spike train) on replay detection. For this study, we avoided any Gaussian smoothing of the posterior probability matrix in our analysis but opted to use relatively large bin sizes for Bayesian decoding (10 cm by 20 ms). It is possible that with smaller bin sizes, replay detection could improve, or yield a different outcome when comparing methodologies. However, as the size of the decoding bin decreases, noise in the decoded trajectory will likely increase or alternatively must be removed using another smoothing method, which in turn may change the false-positive rate of detection. Replay experiments can often vary along many other parameters (e.g. number of neurons, distribution of place fields, length of track, etc.), and it is possible that these differences could also affect the false-positive rate. As such, it is critical for replay studies to independently verify and report an estimated false-positive rate even for the experiments involving only one spatial context.

One caveat associated with this framework is that use of any kind of randomized dataset may potentially underestimate the false-positive rate of replay detection in a way that is biased toward specific shuffling distributions (i.e. place-based or time-based shuffling). In case of a cell-id randomization, each cell’s firing statistics of replay events is preserved, but the relationship between each cell’s spike train and its place fields is randomly swapped, which may be potentially biased more toward place-based shuffling. Generally, we observed that the cell-id randomized dataset did not underestimate the empirically estimated false-positive rates compared to place field shifted dataset and spike train shifted dataset and performed similarly to the cross-experiment shuffled dataset (Figure 1—figure supplement 4). Even for the case of place field shuffling where the spike train shifted dataset resulted in a higher false-positive rates when compared to the cell-id randomized dataset, the difference between cell-id randomized FPR and spike train randomized FPR is much smaller than that between the place field randomized and spike train randomized FPR at an alpha level of 0.05. This suggests that while cell-id randomization is relatively more sensitive to place-based shuffling compared to time-based shuffling, such bias was relatively minor. Taken together, our results suggest that, while imperfect, cell-id randomization (or alternatively a randomization using cross-experiment shuffled place fields) is the relatively more conservative and universally applicable null distribution for empirically estimating false-positive rates given the detection methods used here. Nevertheless, our finding about the performance of different replay detection methods should be interpreted with care. For example, while alpha level adjustment-based empirically estimated FPR allowed us to compare different detection methods, the comparison was made between distributions of candidate replay events rather than based on each individual event. Therefore, two methods with similar performance in terms of track discriminability and proportion of significant events do not necessarily detect the same sets of events as significant. It is also important to note that alpha-level adjustment implemented in this framework is not meant to be a way to optimize detection because it does not directly address method-specific issues and remove true false-positives caused by effects such as edge effects and neuronal spiking statistics.

The use of two linear tracks was critical for our current framework to calculate track discriminability. While this would limit the generality of our approach as most replay studies involved only one context rather than multiple contexts, with modification it is possible that this framework might be adapted to single-track data. For example, track discriminability could theoretically be calculated using a cell-id permutation to create a virtual second track or by comparing the discriminability across different portions of the track (depending on the experimental design), although exploring these options were beyond the scope of our current study.

One key consideration is that due to algorithmic constraints, some sequence detection methods may be biased to detecting certain types of replay trajectories. For instance, a linear fitting method may be more prone to detect linear replay trajectory compared to a sigmoidal-shaped or jumpy replay trajectory. On the other hand, a rank-order correlation analysis of spike trains is more relaxed in terms of the rigidity of the temporal structure, and therefore is more likely to include non-linear events with gaps. As such, there may be other factors to consider beyond track discriminability and the proportion of detected events when choosing a replay detection method. However, it is interesting to note that despite substantial differences in these algorithms used for replay detection, all methods had a similar track discriminability after adjusting the alpha level to a fixed estimated false-positive rate of 5%.

Last but not least, in addition to comparing the performance of different replay quantification strategies, this framework can also be applied to compare replay events detected across different behavioral states or experimental conditions. Previously, differences in replay across conditions have been measured using the rate or number of replay events (Gillespie et al., 2021; Huelin Gorriz et al., 2023; Ólafsdóttir et al., 2015; Xu et al., 2019), or decoding track bias (Carey et al., 2019; Tirole et al., 2022). However, combining these two approaches creates a more comprehensive metric for understanding the difference in replay across different conditions.

All the behavioral and electrophysiological data used in this study were previously described in detail in Tirole et al., 2022.

Analysis code is available at https://github.com/bendor-lab/replay_detection_cross_validation (copy archived at Takigawa, 2024).

1. 1. Tirole M
   2. Huelin M
   3. Takigawa M
   4. Kukovska L
   5. Bendor D

   (2023) Dryad Digital Repository

   Experience-driven rate modulation is reinstated during hippocampal replay.

   https://doi.org/10.5061/dryad.ksn02v76h

1. Software

   1. Asiminas A

   (2020a) Adult hooded rat running, version v1

   Zenodo.

   https://doi.org/10.5281/zenodo.3926237
2. Software

   1. Asiminas A

   (2020b) Adult hooded rat running, version v1

   Zenodo.

   https://doi.org/10.5281/zenodo.3926277
3. 1. Bendor D
   2. Wilson MA

   (2012) Biasing the content of hippocampal replay during sleep

   Nature Neuroscience 15:1439–1444.

   https://doi.org/10.1038/nn.3203

   • PubMed
   • Google Scholar
4. 1. Buzsáki G

   (1989) Two-stage model of memory trace formation: a role for “noisy” brain states

   Neuroscience 31:551–570.

   https://doi.org/10.1016/0306-4522(89)90423-5

   • PubMed
   • Google Scholar
5. 1. Buzsáki G

   (2015) Hippocampal sharp wave-ripple: a cognitive biomarker for episodic memory and planning

   Hippocampus 25:1073–1188.

   https://doi.org/10.1002/hipo.22488

   • PubMed
   • Google Scholar
6. 1. Carey AA
   2. Tanaka Y
   3. van der Meer MAA

   (2019) Reward revaluation biases hippocampal replay content away from the preferred outcome

   Nature Neuroscience 22:1450–1459.

   https://doi.org/10.1038/s41593-019-0464-6

   • PubMed
   • Google Scholar
7. 1. Csicsvari J
   2. O’Neill J
   3. Allen K
   4. Senior T

   (2007) Place-selective firing contributes to the reverse-order reactivation of CA1 pyramidal cells during sharp waves in open-field exploration

   The European Journal of Neuroscience 26:704–716.

   https://doi.org/10.1111/j.1460-9568.2007.05684.x

   • PubMed
   • Google Scholar
8. 1. Davidson TJ
   2. Kloosterman F
   3. Wilson MA

   (2009) Hippocampal replay of extended experience

   Neuron 63:497–507.

   https://doi.org/10.1016/j.neuron.2009.07.027

   • PubMed
   • Google Scholar
9. 1. Diba K
   2. Buzsáki G

   (2007) Forward and reverse hippocampal place-cell sequences during ripples

   Nature Neuroscience 10:1241–1242.

   https://doi.org/10.1038/nn1961

   • PubMed
   • Google Scholar
10. 1. Dragoi G
    2. Tonegawa S

    (2011) Preplay of future place cell sequences by hippocampal cellular assemblies

    Nature 469:397–401.

    https://doi.org/10.1038/nature09633

    • PubMed
    • Google Scholar
11. 1. Eichenbaum H

    (2000) A cortical-hippocampal system for declarative memory

    Nature Reviews. Neuroscience 1:41–50.

    https://doi.org/10.1038/35036213

    • PubMed
    • Google Scholar
12. 1. Foster DJ
    2. Wilson MA

    (2006) Reverse replay of behavioural sequences in hippocampal place cells during the awake state

    Nature 440:680–683.

    https://doi.org/10.1038/nature04587

    • PubMed
    • Google Scholar
13. 1. Foster DJ

    (2017) Replay comes of age

    Annual Review of Neuroscience 40:581–602.

    https://doi.org/10.1146/annurev-neuro-072116-031538

    • PubMed
    • Google Scholar
14. 1. Frankland PW
    2. Bontempi B

    (2005) The organization of recent and remote memories

    Nature Reviews. Neuroscience 6:119–130.

    https://doi.org/10.1038/nrn1607

    • PubMed
    • Google Scholar
15. 1. Gillespie AK
    2. Astudillo Maya DA
    3. Denovellis EL
    4. Liu DF
    5. Kastner DB
    6. Coulter ME
    7. Roumis DK
    8. Eden UT
    9. Frank LM

    (2021) Hippocampal replay reflects specific past experiences rather than a plan for subsequent choice

    Neuron 109:3149–3163.

    https://doi.org/10.1016/j.neuron.2021.07.029

    • PubMed
    • Google Scholar
16. 1. Gomperts SN
    2. Kloosterman F
    3. Wilson MA

    (2015) VTA neurons coordinate with the hippocampal reactivation of spatial experience

    eLife 4:e05360.

    https://doi.org/10.7554/eLife.05360

    • PubMed
    • Google Scholar
17. 1. Grosmark AD
    2. Buzsáki G

    (2016) Diversity in neural firing dynamics supports both rigid and learned hippocampal sequences

    Science 351:1440–1443.

    https://doi.org/10.1126/science.aad1935

    • PubMed
    • Google Scholar
18. 1. Huelin Gorriz M
    2. Takigawa M
    3. Bendor D

    (2023) The role of experience in prioritizing hippocampal replay

    Nature Communications 14:8157.

    https://doi.org/10.1038/s41467-023-43939-z

    • PubMed
    • Google Scholar
19. 1. Ji D
    2. Wilson MA

    (2007) Coordinated memory replay in the visual cortex and hippocampus during sleep

    Nature Neuroscience 10:100–107.

    https://doi.org/10.1038/nn1825

    • PubMed
    • Google Scholar
20. 1. Karlsson MP
    2. Frank LM

    (2009) Awake replay of remote experiences in the hippocampus

    Nature Neuroscience 12:913–918.

    https://doi.org/10.1038/nn.2344

    • PubMed
    • Google Scholar
21. 1. Klinzing JG
    2. Niethard N
    3. Born J

    (2019) Mechanisms of systems memory consolidation during sleep

    Nature Neuroscience 22:1598–1610.

    https://doi.org/10.1038/s41593-019-0467-3

    • PubMed
    • Google Scholar
22. 1. Lee AK
    2. Wilson MA

    (2002) Memory of sequential experience in the hippocampus during slow wave sleep

    Neuron 36:1183–1194.

    https://doi.org/10.1016/s0896-6273(02)01096-6

    • PubMed
    • Google Scholar
23. 1. Lewis PA
    2. Bendor D

    (2019) How targeted memory reactivation promotes the selective strengthening of memories in sleep

    Current Biology 29:R906–R912.

    https://doi.org/10.1016/j.cub.2019.08.019

    • PubMed
    • Google Scholar
24. 1. Muessig L
    2. Lasek M
    3. Varsavsky I
    4. Cacucci F
    5. Wills TJ

    (2019) Coordinated emergence of hippocampal replay and theta sequences during post-natal development

    Current Biology 29:834–840.

    https://doi.org/10.1016/j.cub.2019.01.005

    • PubMed
    • Google Scholar
25. 1. O’Keefe J
    2. Dostrovsky J

    (1971) The hippocampus as a spatial map: preliminary evidence from unit activity in the freely-moving rat

    Brain Research 34:171–175.

    https://doi.org/10.1016/0006-8993(71)90358-1

    • PubMed
    • Google Scholar
26. 1. Ólafsdóttir HF
    2. Barry C
    3. Saleem AB
    4. Hassabis D
    5. Spiers HJ

    (2015) Hippocampal place cells construct reward related sequences through unexplored space

    eLife 4:e06063.

    https://doi.org/10.7554/eLife.06063

    • PubMed
    • Google Scholar
27. 1. Ólafsdóttir HF
    2. Carpenter F
    3. Barry C

    (2016) Coordinated grid and place cell replay during rest

    Nature Neuroscience 19:792–794.

    https://doi.org/10.1038/nn.4291

    • PubMed
    • Google Scholar
28. 1. Ólafsdóttir HF
    2. Carpenter F
    3. Barry C

    (2017) Task demands predict a dynamic switch in the content of awake hippocampal replay

    Neuron 96:925–935.

    https://doi.org/10.1016/j.neuron.2017.09.035

    • PubMed
    • Google Scholar
29. 1. Ólafsdóttir HF
    2. Bush D
    3. Barry C

    (2018) The role of hippocampal replay in memory and planning

    Current Biology 28:R37–R50.

    https://doi.org/10.1016/j.cub.2017.10.073

    • PubMed
    • Google Scholar
30. 1. Pfeiffer BE
    2. Foster DJ

    (2013) Hippocampal place-cell sequences depict future paths to remembered goals

    Nature 497:74–79.

    https://doi.org/10.1038/nature12112

    • PubMed
    • Google Scholar
31. 1. Silva D
    2. Feng T
    3. Foster DJ

    (2015) Trajectory events across hippocampal place cells require previous experience

    Nature Neuroscience 18:1772–1779.

    https://doi.org/10.1038/nn.4151

    • PubMed
    • Google Scholar
32. 1. Squire LR

    (1992) Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans

    Psychological Review 99:195–231.

    https://doi.org/10.1037/0033-295x.99.2.195

    • PubMed
    • Google Scholar
33. Software

    1. Takigawa M

    (2024) Replay_detection_cross_validation, version swh:1:rev:754a9efe01eae551a8e4766f1137c38b1365b2aa

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:815c87241f246593b122a6e6df25a477162fd408;origin=https://github.com/bendor-lab/replay_detection_cross_validation;visit=swh:1:snp:c12aa78007454aed86fc1c2f4f382eeb6f117b2b;anchor=swh:1:rev:754a9efe01eae551a8e4766f1137c38b1365b2aa
34. 1. Tingley D
    2. Peyrache A

    (2020) On the methods for reactivation and replay analysis

    Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 375:20190231.

    https://doi.org/10.1098/rstb.2019.0231

    • PubMed
    • Google Scholar
35. 1. Tirole M
    2. Huelin Gorriz M
    3. Takigawa M
    4. Kukovska L
    5. Bendor D

    (2022) Experience-driven rate modulation is reinstated during hippocampal replay

    eLife 11:e79031.

    https://doi.org/10.7554/eLife.79031

    • PubMed
    • Google Scholar
36. 1. Todorova R
    2. Zugaro M

    (2019) Isolated cortical computations during delta waves support memory consolidation

    Science 366:377–381.

    https://doi.org/10.1126/science.aay0616

    • PubMed
    • Google Scholar
37. 1. van der Meer MAA
    2. Carey AA
    3. Tanaka Y

    (2017) Optimizing for generalization in the decoding of internally generated activity in the hippocampus

    Hippocampus 27:580–595.

    https://doi.org/10.1002/hipo.22714

    • PubMed
    • Google Scholar
38. 1. van der Meer MAA
    2. Kemere C
    3. Diba K

    (2020) Progress and issues in second-order analysis of hippocampal replay

    Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 375:20190238.

    https://doi.org/10.1098/rstb.2019.0238

    • PubMed
    • Google Scholar
39. 1. Widloski J
    2. Foster DJ

    (2022) Flexible rerouting of hippocampal replay sequences around changing barriers in the absence of global place field remapping

    Neuron 110:1547–1558.

    https://doi.org/10.1016/j.neuron.2022.02.002

    • PubMed
    • Google Scholar
40. 1. Wilson MA
    2. McNaughton BL

    (1994) Reactivation of hippocampal ensemble memories during sleep

    Science 265:676–679.

    https://doi.org/10.1126/science.8036517

    • PubMed
    • Google Scholar
41. 1. Xu H
    2. Baracskay P
    3. O’Neill J
    4. Csicsvari J

    (2019) Assembly responses of hippocampal CA1 place cells predict learned behavior in goal-directed spatial tasks on the radial eight-arm maze

    Neuron 101:119–132.

    https://doi.org/10.1016/j.neuron.2018.11.015

    • PubMed
    • Google Scholar
42. 1. Zhang K
    2. Ginzburg I
    3. McNaughton BL
    4. Sejnowski TJ

    (1998) Interpreting neuronal population activity by reconstruction: unified framework with application to hippocampal place cells

    Journal of Neurophysiology 79:1017–1044.

    https://doi.org/10.1152/jn.1998.79.2.1017

    • PubMed
    • Google Scholar

Author details

1. Masahiro Takigawa

   Institute of Behavioural Neuroscience (IBN), University College London (UCL), London, United Kingdom

   Contribution

   Conceptualization, Software, Formal analysis, Funding acquisition, Visualization, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   Masahiro.takigawa.17@ucl.ac.uk

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0162-9017

2. Marta Huelin Gorriz

   Institute of Behavioural Neuroscience (IBN), University College London (UCL), London, United Kingdom

   Contribution

   Software, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0281-0627

3. Margot Tirole

   Institute of Behavioural Neuroscience (IBN), University College London (UCL), London, United Kingdom

   Contribution

   Software, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-0674-6690

4. Daniel Bendor

   Institute of Behavioural Neuroscience (IBN), University College London (UCL), London, United Kingdom

   Contribution

   Conceptualization, Software, Supervision, Funding acquisition, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   d.bendor@ucl.ac.uk

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-6621-793X

• 483

  views

• 61

  downloads

• 0

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.