The permutation test, perfect match concept, and permute-match tests. (A) The permutation test. Circles indicate the average correlation obtained with (grey) or without (pink) trial swapping. Here, N≥ is the number of average correlations (including the original correlation) that are equal to or larger than the original correlation. pperm is then N≥/n! because n! is the total number of average correlations, including the original. (B) Illustration of a Y -perfect match, represented as a table of correlations wherein each diagonal entry is the greatest in its own row. Each line with a “>” or “<” symbol denotes a required ordering relationship between the two numbers at either end of the line. Note that this example has a Y -perfect match, but not an X-perfect match. For an X-perfect match, each diagonal entry would need to be the greatest in its own column. (C) If an X-or Y -perfect match occurs, then the original correlation, ρorig is always greater than any shuffled correlation ρshfl. Top: If a Y -perfect match occurs, then each Yi gives the greatest correlation when it s paired with the Xi from the same trial. A shuffled correlation ρshfl pairs Yis from some trials with Xjs from different trials, thereby reducing the correlation. Lines with symbols “>“, “<“, and “=” denote comparisons between terms. Bottom: If an X-perfect match occurs, a similar argument can be applied. (D) Since a perfect match ensures that the original correlation is greater than any shuffled correlation, a perfect match also ensures that pperm takes on its lowest possible value of 1/n!. (E) The simultaneous permute-match(X;Y) test. (F) The sequential permute-match(X▸Y) test; note that the permute-match(Y ▸X) test is defined analogously.

Statistical power of various permutation test variants as illustrated using a simple nonstationary system. (A) Summary of test power as a function of the significance level α and the number of replicates n. (B, C) System equations and example dynamics. The processes X and Y are given by a linear trend with additive noise on a time grid t = 1, 2, …, 100. The noise terms ϵX,i(t) and ϵY,i(t) are drawn from a bivariate normal distribution with a mean of 0, variance of 1, and covariance of rX,Y. The chart shows an example pair of time series where X and Y are dependent (rX,Y = 0.3). (C) Statistical power of the permutation test and various permute-match tests as a function of the replicate number n, significance level α, and strength of dependence rX,Y. Power was calculated from 5000 simulations at each value of rX,Y between rX,Y = 0 and 0.54 in steps of size 0.01. We chose the Pearson correlation coefficient as our correlation function ρ.

A sequential permute-match test detects dependence between average individual speed and circular concentration in small groups of zebrafish. (A) Time series of average individual speed and circular concentration for the three replicate videos. Black curves show original time series, and red curves show a 30-second moving average. Average individual speed appears to decrease with time in all trials. All 12 time series (2 variables × 3 trials × 2 smoothing conditions) were deemed nonstationary by a Kwiatkowski– Phillips–Schmidt–Shin (KPSS; [37]) test (p < 0.01). Note that the KPSS test seeks to reject a stationary null hypothesis in contrast to other common tests, whose null hypotheses are nonstationary. (B) Pearson correlation between circular concentration and average individual speed, both within trials and between trials, using the full 10 minutes of data. Table entries are shaded by correlation. We observe a speed-perfect match (and a circular concentration-perfect match), and thus detect dependence with p = 1/33 ≈ 0.04. (C) A parametric test also detects dependence. As a parametric alternative, for each trial we averaged values of speed and circular concentration over the full 10 minutes (e.g. the scatter plot shown, where each point is one trial). The sample Pearson correlation of the time-averaged variables is 0.99996, and a one-tailed test of significance gives p ≈ 0.003. (D) Permute-match tests detected a significant correlation between speed and circular concentration more consistently than the parametric test. We truncated time series to different lengths starting from the first frame, and compared the parametric test with the two possible permute-match tests. Length was varied between 20 seconds and 10 minutes in 20-second increments, with each increment representing 640 frames since the frame rate is 32 frames per second. We used the Pearson correlation (not its magnitude) in the permute-match tests since the alternative hypothesis is that the correlation is positive, not merely nonzero.

The procedure of Eq A-10 produces a false positive rate that generally exceeds α, and depends on the relationship between tests X and Y. This relationship X and Y can be quantified by P′, the probability that test Y is significant (i.e. 2pY ≤α) but test X is not significant (i.e. pX > α). The false positive range is shown over the full range of P′, from 0 to α/2. The only way for the overall procedure of Eq A-10 to be valid is when P′ = 0, where tests X and Y are so tightly coupled that a significant result in test Y deterministically ensures that test X is also significant. Other than this edge case, the false positive rate of the overall procedure will exceed α.

Joint and marginal probabilities of the outcomes of tests X and Y, assuming that pX and pY both follow Unif(0, 1). The parameter P′ is the probability that test Y is significant but test X is not.

Statistical power of permute-match tests and the permutation test in a nonlinear and nonstationary system. (A) System equations. (B) Example dynamics. The processes X and Y are given by a coupled logistic map on a time grid t = 1, 2, …, 100. The system is nonstationary because the parameter r(t) varies with time from 3.72 when t = 1 to 3.82 when t = 99. To see the nonstationarity clearly, we plot Xi(t) against Xi(t + 1) and color points by time, showing that as time progresses, there is an upward drift in the parabola that the points lie on. To better show the trend, 10 replicates are shown simultaneously in this chart. (D) Statistical power of the permutation test and permute-match tests as a function of the number of replicates, the significance level, and rX,Y. Power was calculated from 5000 simulations at each value of rX,Y between rX,Y = 0 and rX,Y = 0.2 in steps of size 0.005. The correlation statistic (ρ) was cross-map skill, which is known to readily detect dependence between X and Y in this system [45]. For the cross-map skill calculation, we used Y to estimate X, which corresponds to a scenario where the data analyst hypothesizes that X influences Y. Cross-map skill requires two parameters, the embedding dimension and the embedding lag, and these were set to 2 and 1 respectively following prior works that used the logistic map for benchmarking [45, 46].