You do not know anything until you know why you know it.

Clovis Andersen, The Principles of Private Detection (McCall Smith, 2007), cited in Sokal, 2010.

Introduction

Scientific models are explanations of reality (Shmueli, 2010; Frigg and Hartmann, 2020). Models come in many forms, from sentences to equations, and in many kinds, from hypotheses to theories. All models are false, but some models are truer than others (Mizrahi, 2020). Specifically, all else being equal, models that are more explanatorily successful — that explain the data more accurately or with fewer assumptions — are likely to be truer than rival models (Appendix 1).

Efforts to find truer models drive scientific progress but command relatively little neuroscientific attention. Neuroscience devotes greater efforts to produce better data or more replicable analyses (Frégnac, 2017). A study by Jonas and Kording, 2017 implicitly critiqued this imbalance of effort. The study showed that popular neuroscientific analyses of ideal data cannot explain the workings of a computer chip, a toy model of the nervous system. The study implied, in this way, that neuroscience must devote greater efforts to find truer models.

Science finds truer and truer models relative to stronger and stronger rival models. By contrast, many analyses in neuroscience test new speculative models against weak null models. Some of these analyses use circular reasoning to redundantly explain existing knowledge. These circular analyses of knowledge violate the principle of parsimony and, in this way, accept models that are less true relative to the strongest rival models. Here, I discuss the nature and prevalence of this problem in systems and network neuroscience. I show that the problem can confound key results and estimate that it is common in the network-neuroscience literature.

I suggest that studies can reduce this problem in three main ways. First, they can limit the use of speculative evidence. Second, they can integrate all important existing knowledge into benchmark models. Third, they can rigorously test the significance of proposed discoveries against these models. Together, these steps can reduce circular analyses, formalize existing knowledge, and benchmark future progress.

Much of the following discussion stresses the importance of unambiguous definitions. Accordingly, Table 1 defines the use of several potentially ambiguous technical terms.

Table 1

Term	Definition
Principle of parsimony (Occam’s razor)	An assertion that all else being equal, models with fewer redundant features are likely to be truer than rival models (Baker, 2022). This assertion reflects an objective preference for parsimony rather than a subjective preference for simplicity or elegance. In this way, and contrary to misconception, the principle of parsimony does not imply that reality, or its truest models, are simple or elegant.
Trueness (bias)	Distance between expected and true estimates of model parameters (ISO, 1994). True values of model parameters are typically inaccessible, and trueness (bias) can therefore be defined only in relative terms. The principle of parsimony asserts that all else being equal, models with fewer redundant features have truer (less biased) parameter estimates relative to rival models.
Precision (variance)	Expected distance between repeated estimates of model parameters (ISO, 1994). Precision (variance) does not require knowledge of the true values of model parameters and can therefore be defined in absolute terms. The problem of irreplicable results (Ioannidis, 2005) is primarily a problem of precision (variance).
Circular analysis	An analysis that first tests a model in a way that almost invariably accepts the model and then accepts the model on the basis of this test. This definition includes circular analyses of knowledge that accept overspecified models or redundant (less true) explanations. It also includes circular analyses of noise that accept overfitted models or irreplicable (less precise) explanations (Kriegeskorte et al., 2009).
Neural circuits or brain networks	Groups of connected neurons or brain regions that mediate function. This definition does not intend to make analogies between groups of neurons or brain regions, and electronic circuits or artificial neural networks (Rubinov, 2015).
Function	Behavior and other action that helps animals to survive and reproduce (Roux, 2014). This definition excludes physiological phenomena that lack such useful action.
Structure	Anatomical or physiological organization. This definition encompasses all physiological phenomena, including phenomena that lack known function.
Development	Formation of structure before and after birth. This definition includes plasticity and therefore encompasses learning and memory.

General definitions

Analyses of complex datasets are vulnerable to distortions by extraneous features. Such distortions may include corruption by noise or confounding by existing knowledge. Statistical science, machine learning, and other fields have developed rigorous tests to mitigate the risk of these distortions. Analyses of complex datasets that neglect such tests, however, will almost invariably be distorted by extraneous features to some extent.

These distortions can generally lead to inflated agreement between model and data and to inappropriate model acceptance on the basis of this inflated agreement. The nature of individual distortions, however, will ultimately determine the individual consequences of this problem. On the one hand, corruption of analyses by noise can lead to the well-known problem of model overfitting and to irreplicable explanations (Kriegeskorte et al., 2009; Vul et al., 2009). On the other hand, confounding of analyses by existing knowledge can lead to a distinct, and less well-known, problem of model overspecification and to redundant explanations.

This work describes analyses that neglect to test speculative models against existing knowledge and that consequently accept overspecified models and redundant explanations. This section first defines the nature of this problem and then outlines a general solution.

Toy analogy

We can get an intuition for the problem with a toy analysis of a biological image (Figure 1a). The image is ambiguous, but our existing biological knowledge tells us that it most likely shows a duck — specifically a male duck doing a head-throw, its signature courting move. Sometimes our analyses may neglect such knowledge. This neglect will not make knowledge disappear. Instead, it will inflate the importance of hypotheses redundant with this knowledge.

Figure 1

Download asset Open asset

We may propose, for example, that the image shows a skvader, a type of winged hare (Figure 1b). Our existing knowledge makes this hypothesis redundant — ducks doing head-throws almost always look like skvaders. Our neglect of this knowledge, however, can make the hypothesis seem important. We may accept the hypothesis on the basis of this perceived importance. This acceptance, however, will lead to redundant explanations. We will implicitly “double dip” or explain the same image twice — first as a duck and second as a skvader.

Circular analysis

We can define the problem more formally with three types of models.

Benchmark models (well-specified models). These models represent all important existing knowledge about our phenomenon of interest. They include all benchmark features, features of known importance to this phenomenon, and they exclude all other features. In systems neuroscience, benchmark features often represent existing knowledge about the function, structure, development, and evolution of neural circuits. Distinct phenomena may have distinct benchmark models, and one phenomenon may have several competing benchmark models.

Speculative models. These models represent new hypotheses about some phenomenon of interest. They include one or more speculative features, features of possible but uncertain importance to this phenomenon. Some speculative features may turn out to be redundant with benchmark features. For example, consider the similarity of the human brain and the universe (Figure 1c–d). Both systems have billions of nested, spatially embedded, and interacting elements: neurons and galaxies (Vazza and Feletti, 2020). Let the feature of cosmicity denote the resemblance of a complex system to the universe. The human brain has high cosmicity. A speculative model may propose, on this basis, that brain dynamics resemble cosmic dynamics. Note, however, that brain cosmicity is likely to be redundant with our existing knowledge about the structure of neural circuits.

Strawman models (underspecified models). These models represent weak null hypotheses. They typically exclude the benchmark features with which the speculative features are redundant. In our example, a strawman model excludes the known structure of neural circuits with which cosmicity is redundant.

Circular analyses. These analyses almost invariably accept speculative models against strawman models (Box 1, Appendix 2). They comprise circular analyses of noise and circular analyses of knowledge (Appendix 3). Circular analyses of noise, the focus of previous work (Kriegeskorte et al., 2009), result in acceptance of noisy or irreplicable explanations. By contrast, circular analyses of knowledge, the focus of this work, result in acceptance of redundant explanations. In our example, a circular analysis of knowledge will almost invariably accept the significance of cosmicity against our strawman model. The analysis will be circular because our strawman model excludes the known structure of neural circuits with which cosmicity is redundant.

Box 1

A classification of circular analyses.

General definition (weak evidence of progress)

Circular analyses are analyses that use circular reasoning. These analyses:

1. Test a model in a way that almost invariably accepts the model.

2. Accept the model on the basis of this test.

In general, circular analyses denote weak evidence of progress but do not necessarily preclude progress. In this way, these analyses do not necessarily denote strong evidence of stagnation. These analyses also violate Mayo’s weak-severity requirement of “bad evidence, no test” (Mayo and Spanos, 2011; Mayo, 2018; Appendix 2).

Specific definition (strong evidence of stagnation)

This work describes circular analyses of knowledge. These analyses:

1. Test a speculative model in a way that almost invariably accepts it against a strawman model. Specifically, these analyses test the statistical significance of speculative features in a way that almost invariably shows the significance of these features against a strawman model because:

   1. The speculative features are redundant with one or more benchmark features.

   2. The strawman model excludes the benchmark features with which the speculative features are redundant.

2. Accept the speculative model on the basis of this test.

Circular analyses of knowledge explain the same aspect of the data twice: first, as one or more benchmark features and second, as a speculative feature redundant with these benchmark features. In this way, these analyses necessarily denote strong evidence of stagnation. Note that in principle, the acceptance of redundant explanations may signify regress rather than mere stagnation. In practice, however, the relatively transient nature of many such explanations suggests that stagnation is a more apt description of the problem, cf. “[w]hen we examine the history of favored stories for any particular adaptation, we do not trace a tale of increasing truth as one story replaces the last, but rather a chronicle of shifting fads and fashions.” (Gould, 1978)

Analyses of noise and analyses of knowledge. Previous work has described circular analyses of noise (Kriegeskorte et al., 2009). These analyses have deep similarities with circular analyses of knowledge. Both analyses center on the problem of false discovery and are equivalent in other important respects (Appendix 3).

Redundant explanations (overspecified models). Studies sometimes conclude that speculative features should replace or overturn the benchmark features with which they are redundant. Circular analyses of knowledge cannot support such conclusions because they never test the speculative features against a benchmark model. Such analyses must therefore accept, often implicitly, a model that includes all the existing benchmark features and the redundant speculative features.

In our example, we do not test cosmicity against existing knowledge with which it is redundant and so cannot overturn this existing knowledge. Our analysis implies, therefore, that cosmicity enriches, but does not replace, our existing knowledge. In this way, we must accept the importance of cosmicity and simultaneously accept the importance of existing knowledge with which cosmicity is redundant.

This problem extends to the acceptance of many, potentially countless, speculative models against the same strawman model. Such acceptance implicitly proposes the simultaneous importance of many, potentially countless, redundant features. Moreover, the circular acceptance of one speculative model after another can give an impression of progress even as it leads to stagnation.

Unified analysis

A general solution to this problem centers on significance tests of speculative features against benchmark models (Figure 2). These tests represent unified analyses of existing knowledge and proposed discovery. They form controlled experiments that test the importance of one feature by controlling for the effects of all known confounding features (Sibbald and Roland, 1998; Box 2). They also form a type of severe (model) selection within Mayo’s framework of severe testing (Mayo and Spanos, 2011; Mayo, 2018; Appendix 2). Finally, they parallel controls for model overfitting (Kriegeskorte et al., 2009; Appendix 3).

Figure 2

Download asset Open asset

Box 2

Tests against benchmark models and randomized controlled trials.

Tests against benchmark models have deep similarities with randomized controlled trials, controlled experiments in medical research (Sibbald and Roland, 1998). Randomized controlled trials comprise three main steps. The first step randomly splits a sample of people into a treated group and a control group. The second step gives the treatment to people in the treated group and gives a placebo to people in the control group. The third step compares the medical outcomes of the two groups.

The following list shows that tests against benchmark models (or tests) have essentially the same structure as randomized controlled trials (or trials), even as they differ in implementational details.

• Samples of empirical data (in tests) parallel people in the treated group (in trials).

• Samples of benchmark-model data (in tests) parallel people in the control group (in trials).

• Comparison of test statistics (in tests) parallels comparison of medical outcomes (in trials).

• Maximally random, or unbiased, sampling of benchmark-model data (in tests) parallels maximally random, or unbiased, split into the treated and control groups (in trials). Both approaches allow, in principle, to control for all known (tests) or all possible (trials) confounding explanations.

Despite these similarities, these two types of experiments have one basic difference. Randomized controlled trials can test causality because the treatment always precedes the outcome in time (Siddiqi et al., 2022). By contrast, tests against benchmark models can test non-redundancy but do not test causality unless we have additional information about the temporal precedence of speculative and benchmark features.

In practice, these analyses center on the sampling of data from benchmark-model distributions and on the testing of speculative features against these data. We can describe these analyses in three steps.

First, we can consider a sample of empirical data. The sample could be as small as a single dataset (Figure 2a) or it could be larger. We can compute a speculative feature of interest on this sample and summarize this feature with a test statistic (Figure 2b and e). The empirical test statistic reflects the importance of the corresponding speculative feature. It can also reflect, by extension, the importance of the speculative model that centers on this feature.

Second, we can get many data samples from a benchmark-model distribution (Figure 2c). These samples should match the statistics of all benchmark features but be maximally random in all other respects. We can compute the test statistic on these samples and in this way estimate the null distribution — the distribution of the test statistic under the null hypothesis of existing knowledge (Figure 2d and e).

Third, we can test the significance of the empirical test statistic against this null distribution by estimating the effect size, uncertainty (confidence) interval, and p-value (Mayo and Spanos, 2011). The p-value can reflect the probability that the empirical test statistic does not exceed the benchmark test statistic. In this way, and with appropriate definitions of the test statistic and the benchmark model, the p-value can reflect the probability that our proposed discovery does not transcend existing knowledge.

In our cosmicity example, we can do this analysis in three steps. First, we can define a test statistic of cosmicity and estimate the empirical value of this statistic. Second, we can define a benchmark model that includes all important existing knowledge about the structure of neural circuits. We can then sample data from this model distribution and estimate the null distribution of the test statistic. Third, we can use this null distribution to estimate the effect size, the uncertainty interval, and the p-value, and in this way test the significance of cosmicity against our existing knowledge of neural circuits.

As we discussed above, cosmicity is likely to be redundant with our existing knowledge. This likely redundancy suggests that our result is unlikely to be statistically significant. In this context, a finding of statistical significance can serve as genuine evidence for the importance of cosmicity and, by extension, for the importance of cosmic dynamics to brain function.

Specific examples

Previous work has noted that circular analyses of noise can be “hard to understand, imagine, or predict” and “when it’s hard to see how, it can still be happening” (Kriegeskorte et al., 2009). This section shows that circular analyses of knowledge can often be similarly inconspicuous. It first describes possible examples of these analyses in systems neuroscience and probable examples in network neuroscience. It then walks through the details of the problem with a toy analysis. It finally estimates the prevalence of the problem in the network-neuroscience literature.

Possible circular analyses of knowledge

Systems neuroscience broadly studies the structure and function of interacting groups of neurons or brain regions. The field variously terms these groups assemblies, populations, circuits, systems, or networks. It has acquired considerable, albeit somewhat scattered, knowledge about the structure and function of these groups. It has also proposed many speculative hypotheses that seek to transcend this existing knowledge.

We can show how circular analyses of knowledge can lurk in this environment using the example of the systems neuroscience of (mammalian) vision. In line with our discussion, we can first consider the benchmark, speculative, and strawman models of this phenomenon.

Benchmark model. Systems neuroscience lacks a benchmark model that captures our existing knowledge about the nature and origin of vision (Poggio and Serre, 2013; Golan et al., 2023). Despite this lack of a benchmark model, we know many benchmark features relevant to vision. We know, for example, that the visual system tightly balances the activity of inhibitory and excitatory neurons (Isaacson and Scanziani, 2011). This balance prevents overinhibition and overexcitation and thus allows animals to sense light and not get seizures (Ma et al., 2019). We also know that this balance rests, in part, on the fast-spiking response of inhibitory neurons to excitatory visual stimulation (Sohal, 2016). Finally, we know that vision evolved, in virtually all animals, to support visuo-motor interactions, that is, to help animals interact with their environments through movement (Goodale, 1996; Nilsson, 2021). These basic features do not necessarily form a benchmark model, but they will suffice for our discussion.

Speculative models. Systems neuroscience has many speculative models of vision. Many of these models center on the importance of elegant features and often rest on analogies with other natural and synthetic systems. We can consider three prominent examples of these models.

The first model centers on the importance of internal representations, patterns of neuronal activity that internally represent visual stimuli (Craik, 1943; Hubel and Wiesel, 1959). Studies have proposed that the visual system interprets the meaning of internal representations much like an artificial neural network decodes the nature of input images (Kriegeskorte, 2015; Richards et al., 2019; Cichy and Kaiser, 2019; Figure 1e–f). Despite these intuitions, we have no evidence that patterns of neuronal activity actually denote internal representations (Kenny, 1971; Brette, 2019; Bennett and Hacker, 2022). Moreover, in many cases, we may be able to explain these patterns as substrates of visuo-motor interactions without the need to assume that they internally represent anything at all (Freeman and Skarda, 1990; Cao, 2020; Driscoll et al., 2022).

The second model centers on the importance of gamma oscillations, fast rhythms of neuronal activity that correlate with visual perception (Gray et al., 1989; Burwick, 2014). Studies have proposed that gamma oscillations bind simple visual stimuli into complex perception, much like orchestra conductors weave the sounds of individual musicians into complex music (Singer, 2001; Buzsáki and Draguhn, 2004). Despite these intuitions, we know that gamma oscillations are absent during the perception of some images, and so may not be necessary to bind stimuli into perception (Hermes et al., 2015b; Hermes et al., 2015a). Moreover, in many cases, we may be able to explain these oscillations as the inevitable outcomes of inhibitory responses to visual stimulation without the need to assume that they bind anything at all (Ray and Maunsell, 2015; Singer, 2018).

The third model centers on the importance of neural criticality, collective neuronal activity that balances on the edge of order and disorder. Studies have proposed that criticality optimizes our sensitivity to visual stimuli, much like the critical (neither shallow nor steep) angle of a sand pile optimizes its responsiveness to tactile stimuli (Shew et al., 2009; Shew and Plenz, 2013). Despite these intuitions, we know that signatures of criticality can occur in the absence of any visual stimuli and so may not necessarily be related to optimized visual sensation (Fontenele et al., 2019; Destexhe and Touboul, 2021). Moreover, in many cases, we may be able to explain these signatures as inevitable outcomes of balanced inhibitory and excitatory activity without the need to assume that they optimize anything at all (Nanda et al., 2023).

Strawman models. We cannot summarize the full range of null models in the expansive literature of representations, oscillations, and criticality. We can still do justice to this literature, however, by considering some of its strongest models. One such model can test the significance of representations against correlations of neuronal activity across space and time (Elsayed and Cunningham, 2017). Another model can test the significance of oscillations against non-oscillatory activity of similar amplitude (Donoghue et al., 2022). A third model can test the significance of critical neuronal activity against mimicking non-critical (lognormal) phenomena (Buzsáki and Mizuseki, 2014). Together, all these models can test representations, oscillations, and criticality against important confounders. Despite this, none of these models test these speculative features against the benchmark features with which they may be redundant.

Circular analyses and redundant explanations. Tests against strawman models often accept the importance of representations, oscillations, and criticality. Separately, these tests cannot reject the importance of benchmark features with which these speculative features may be redundant. It follows that these tests may implicitly explain the same aspects of brain activity twice — first as a basic benchmark feature and second as a redundant speculative feature. In the study of vision, these analyses may therefore conclude the simultaneous importance of:

1. Visuo-motor interactions and internal representations possibly redundant with these interactions.

2. Inhibitory responses to stimulation and gamma oscillations possibly redundant with these responses.

3. Balance of inhibition and excitation and critical activity possibly redundant with this balance.

Individually, these analyses accept simple or elegant models. Collectively, however, they may accept a needlessly complicated model that assumes the simultaneous importance of several redundant features.

Probable circular analyses of knowledge

Many parts of systems neuroscience, such as the study of vision, lack well-defined benchmark models or the ability to test speculative models against these benchmarks. These limitations make it hard to show the presence of circular analyses of knowledge, even when they exist.

Some parts of systems neuroscience, however, have relatively well-defined benchmark models and the ability to test speculative models against these benchmarks. These strengths make it possible to show the presence of circular analyses of knowledge when they exist. Here, we can describe the probable presence of such analyses in network neuroscience.

Network neuroscience is a subfield of systems neuroscience that studies the structure and function of extensive, including whole-brain, networks (Bassett and Sporns, 2017). Nodes in these networks typically denote cells or regions, while links typically denote synapses, axonal projections, or activity correlations. We can show probable circular analyses in this field using the example of the network neuroscience of (mammalian) cortex. In line with our previous discussion, we can first consider the benchmark, speculative, and strawman models of this structure.

Benchmark model. We have considerable knowledge of evolution, development, structure, and function of cortical networks. First, evolutionary analyses of extensive mapping studies suggest that essentially all mammals share a common cortical blueprint (Kaas, 1995; Krubitzer, 1995; Figure 3a). Second, the commonality of this blueprint likely stems from strongly conserved developmental processes. These processes include an initial establishment of spatial concentration gradients of developmental molecules and a subsequent discretization of these gradients (Figure 3b–c). Third, signatures of these developmental processes show through in the structure of the adult cortex. To a first approximation, this structure reflects a gradual transition along the cortical sheet (Figure 3d):

• from a relatively well-delineated, clustered, and poorly connected sensory-motor cortex.

• to a relatively ill-delineated, distributed, and highly connected association cortex.

(The sensory-motor cortex is well-delineated in large part because it comprises cortical areas that form spatial mappings of entire sensory or motor fields. For example, the primary somatosensory area comprises a spatial mapping of all body parts that can receive somatic input. By contrast, the association cortex is ill-delineated in large part because it lacks areas that form similarly clear mappings of complete sensory or motor fields [Buckner and Krienen, 2013; Patel et al., 2014].)

Fourth, this cortical structure constrains known cortical function. Specifically, a gradual transition from a relatively well-delineated sensory-motor cortex to a relatively ill-delineated association cortex reflects a corresponding transition from relatively well-defined sensory-motor function to relatively ambiguous cognitive function (Bayne et al., 2019).

Figure 3

Download asset Open asset

Network neuroscience has a well-known model that captures the basic features of this cortical blueprint (Sporns, 2013). This model includes two types of benchmark features. First, it includes network modules (clusters) that capture the clustered sensory-motor cortex. Second, it includes node connectivity (number of connections) that captures the gradual transition from the poorly connected sensory-motor cortex to the well-connected association cortex. We can adopt this basic benchmark model for our subsequent discussion.

Speculative models. Speculative models in network neuroscience broadly resemble other speculative models in systems neuroscience. These models center on the importance of elegant features and often rest on analogies with other natural and synthetic systems, such as metabolic, transport, and friendship networks (Barabási, 2016; Figure 1g–h). In contrast to broader systems neuroscience, however, speculative features in network neuroscience are often more clearly redundant with benchmark features. We can show this redundancy in three speculative models that reflect some of the best-known results in network neuroscience.

The first model centers on the importance of small-world structure (Stephan et al., 2000; Achard et al., 2006). This structure denotes the simultaneous presence of many network triangles (triplets of fully connected nodes) and many network shortcuts (connections between different network parts). Studies have proposed that small-world cortical structure optimizes the competing demands of functional segregation and integration (Sporns and Zwi, 2004). We also know, however, that this structure is redundant with connected sensory-motor modules: “modular systems are small-world but not all small-world systems are modular” (Meunier et al., 2010).

The second model centers on the importance of cores or clubs (Hagmann et al., 2008; Zamora-López et al., 2010; van den Heuvel and Sporns, 2011). These structures denote groups of highly connected nodes. Studies have proposed that cores or clubs of the association cortex form the backbone of functional integration and may underpin the global workspace, a theoretical substrate of consciousness (Griffa and van den Heuvel, 2018). We also know, however, that these structures are redundant with sensory-motor modules and highly connected association nodes (hubs): “clubs are structural byproducts of modules and hubs” (Rubinov, 2016).

The third model centers on the importance of node controllability (Tang et al., 2012; Gu et al., 2015). High-control nodes in dynamical systems mediate switches between network activity (system states). Studies have proposed that high-control cortical nodes may support internal cognitive control and may serve as levers for external cortical control (Tang and Bassett, 2018). We also know, however, that node controllability is roughly equivalent with node connectivity (degree) (Tu et al., 2018) or related features (Patankar et al., 2020): “a strong […] correlation between node degree and average controllability is mathematically expected” (Gu et al., 2015).

Strawman models. Studies of small worlds, cores/clubs, and controllability use a relatively limited set of null models. First, tests of small worlds and cores/clubs tend to follow the broader network-science literature and use null models that include node connectivity but not network modules (Watts and Strogatz, 1998; Colizza et al., 2006). Second, many tests of controllability use abstract null models that lack node connectivity or network modules (Pasqualetti et al., 2019). Third, many studies also use null models that include the empirical decay of connectivity with spatial distance (Markov et al., 2013). These spatial models can account for much variance in the data and are perhaps the strongest network-neuroscience null models in common use today (Kaiser and Hilgetag, 2006). Despite these strengths, these models lack node connectivity or network modules and cannot compete with benchmark models that include these features (Rubinov, 2016).

Circular analyses and redundant explanations. Tests of small worlds, cores/clubs, and controllability against strawman models will almost invariably accept the importance of these speculative features. Separately, these tests cannot reject the importance of benchmark features with which these speculative features are redundant. It follows that these circular analyses implicitly explain the same aspects of network structure twice: first as a basic benchmark feature, and second as a redundant speculative feature. Individually, these analyses accept simple or elegant models. Collectively, however, they accept a needlessly complicated model that assumes the simultaneous importance of sensory-motor modules, highly connected association nodes, small worlds, cores/clubs, and controllability.

Walkthrough circular analysis of knowledge

We can show the details of this problem with a walkthrough analysis of a toy cortical network. This network has an accentuated transition from clustered to distributed cortical connectivity (Figure 4a–b, left). We can propose a speculative model of this network that centers on a toy feature of a controllable core. This hybrid feature represents a core of cortical regions whose activity can be induced with relatively little stimulation. Theory suggests that this controllable core may support a stable state of cortical activity and thus play an important role in cortical function. Despite these considerations, the existence and importance of this feature remain speculative without tests against a benchmark model.

Figure 4

Download asset Open asset

We can test this feature against a benchmark model in three steps. First, we can define a test statistic that reflects the importance of this feature. In our example, we can define this statistic to be the core density of controllable network nodes (Figure 4a, right). Second, we can compute the value of this statistic on empirical and benchmark-model data (Figure 4b). Third, we can use these values to quantify the effect size, uncertainty interval, and p-value. In our analysis, the empirical test statistic is 3.02, while the median [95% uncertainty interval] benchmark-test statistic is 3.00 [2.87, 3.12] (arbitrary units). The corresponding effect size of 0.02 [−0.11, 0.15] and p=0.36 (Figure 4b, right) suggest that the empirical test statistic is not significant against benchmark-model data. This analysis suggests that the controllable core is redundant with our existing knowledge.

Separately, we can test the significance of a controllable core against a strawman model. (Figure 4c). In our analysis, the strawman-model statistic is 2.72 [2.62, 2.88]. The corresponding effect size of 0.30 [0.14, 0.40] and p<0.01 (Figure 4c, right) suggest a rejection of this strawman model. This rejection is circular because the strawman model excludes the benchmark feature with which the controllable core is redundant.

Prevalence of probable circular analyses of knowledge

I quantified the fraction and number of probable circular analyses of knowledge in the network-neuroscience literature. I did this by evaluating network-neuroscience studies published during five recent years in ten journals. Appendix 4 describes the details of this evaluation.

This evaluation shows that 56% of evaluated studies had at least one circular analysis of knowledge. A simple extrapolation suggests that this problem may have affected more than three thousand original studies published over the last decade. This extrapolation is necessarily a rough estimate. It may be upwardly biased if my sample is unrepresentative of the broader literature or downwardly biased if my search criteria missed other affected articles. Despite these limitations, this extrapolation forms a useful indicator of the magnitude of this problem in the literature.

I did not try to assess the effects of this problem on individual results. These effects will depend on the aims and conclusions of individual studies. For example, circular analyses of knowledge in some studies may be tangential to the main results and may not affect the main conclusions. Separately, circular analyses in other studies may make the main results seem falsely novel or important and, in this way, may severely distort the main conclusions. Overall, I agree with a previous similar evaluation of the literature (Kriegeskorte et al., 2009) that such effects should be assessed through systematic community efforts.

To facilitate these efforts, I created a semi-automated analysis pipeline that downloads and curates all published studies that match some specified search criteria (Appendix 4). The curation includes the extraction of the Methods and Results sections and the highlighting of possible descriptions of benchmark, speculative, or strawman models. This basic curation cannot replace the careful evaluation of individual articles, but it may help to make such an evaluation standardized and more objective.

Speculative evidence

The commonness of circular analyses of knowledge may reflect, in part, the intuitive importance of many speculative models. This importance often rests on the misleading suggestiveness of speculative evidence. The ability to spot such evidence can help to shift the focus from speculative intuitions to rigorous tests and, in this way, alleviate much of this problem in the literature.

This section discusses how suggestive terminology, suggestive structure, and suggestive narratives can all falsely signal the importance of speculative features. This discussion aligns with similar perspectives in neuroscience (Krakauer et al., 2017), machine learning (Lipton and Steinhardt, 2019), and psychology (Yarkoni, 2020).

Suggestive terminology: Deepities

The term deepity denotes a word or phrase that has two distinct meanings (Dennett, 2013). The first meaning is direct and undisputed but bland, while the second is profound but indirect and speculative. Deepities do damage when they lead us to conflate the two meanings and, in this way, make speculative or redundant features seem well-supported.

Many bedrock terms or ideas in systems neuroscience are deepities because they conflate facts with speculations (Table 2). Here, we can show this conflation using three especially consequential terms: function, emergence, and significance. We can do so using a toy example of “lub-dub” heart sounds, features that arise as byproducts of turbulent blood flow.

Table 2

Deepity	Direct meaning	Implicit allusion
Neural computation (Churchland and Sejnowski, 2016)	Transformation of sensory input to behavioral output.	Computer-like transformation of sensory input to behavioral output.
Neural representation, code, or information (Baker et al., 2022; Brette, 2019; Nizami, 2019)	Patterns of neuronal activity that correlate with, or change in response to, sensory input.	Internal representations or encodings of information about the external world.
Neural networks (Bowers et al., 2022)	Artificial neural networks (machine-learning models).	Biological neural networks.
Necessity and sufficiency (Yoshihara and Yoshihara, 2018)	The induction or suppression of behavior through stimulation or inhibition of neural substrate.	Logical equivalence between behavior and neural substrate.
Functional connectivity (Reid et al., 2019)	Correlated neural activity.	Neural connectivity that causes function.
Complexity (Merker et al., 2022)	Patterns of neural structure that are neither ordered nor disordered.	Patterns of neural structure that are fundamentally important.
Motifs	Repeating patterns of brain-network connectivity.	Motifs of neural computation.
Efficiency	Communication between pairs of brain nodes via algorithmic sequences of connections.	Efficiency of neural communication.
Modularity	Propensity of brain networks to be divided into clusters.	Propensity of brain networks to be robust or evolvable.
Flexibility	Propensity for brain nodes to dynamically switch their cluster affiliations.	Propensity for cognitive flexibility.
The brain is a network, like many other natural and synthetic systems.	The brain consists of connected elements, like many other natural and synthetic systems.	The brain shares functional network principles with many natural and synthetic systems.
Brain disorders are disconnection syndromes.	Brain disorders are correlated with brain-network abnormalities.	Brain disorders are caused by brain-network abnormalities.

First, function can denote physiological activity and also signal functional utility (Roux, 2014). The conflation of these two meanings may falsely attribute utility to all physiological phenomena. The heart pumps blood and makes lub-dub sounds, but only one of these actions is useful.

Second, emergent phenomena can denote higher-order structures in complex systems and also signal the importance of these structures (Bedau, 1997). The conflation of these two meanings may falsely attribute functional importance to higher-order structures. The structure of turbulent blood flow is emergent, but this flow plays no important role in heart function.

Third, significance can denote the rejection of a null hypothesis and also signal scientific importance (Wasserstein and Lazar, 2016). The conflation of these two meanings may falsely attribute importance to statistically significant features, especially if these features are also functional and emergent. In practice, the importance of a statistically significant result is strongly tied to the nature of the null hypothesis. A weak null hypothesis may propose, for example, that heart sounds are equally loud in still and beating hearts. We will always reject this null hypothesis, but such rejection will tell us little about the importance of heart sounds.

Collectively, the use of deepities can make speculative features seem useful or important. Moreover, the ability to fall back on the direct meanings of deepities in response to criticism, and to promote their implicit allusions at other times, can make deepities easy to defend and thus hard to eliminate. (This defense of deepities is known as “motte and bailey”, by analogy with a defense of a medieval castle [Shackel, 2005]. The motte is a hill with a tower — it is easily defensible but not particularly enjoyable to spend time in. The bailey is an outside court — it is enjoyable but not particularly defensible. The motte-and-bailey defense denotes a retreat to the motte in response to attacks and enjoyment of the bailey during more peaceful times.)

Suggestive structure: Spandrels

In architecture, spandrels denote triangular spaces of building arches (Figure 5a). These spaces arise as byproducts of the contours of the arch, but their intricate decoration may suggest that they have important (decorative) function. In biology, spandrels are phenotypes that have intricate and similarly suggestive structure (Gould and Lewontin, 1979). For example, the intricate structure of turbulent lub-dub flow, and the ability of this flow to predict heart activity and physical exertion, may all suggest that lub-dub sounds play an important role in heart function. The intricate structure and predictive success of many features in systems neuroscience may likewise suggest that these features play an important role in brain function (Figure 5b–c).

Figure 5

Download asset Open asset

The concept of spandrels helps to show the value of tests against benchmark models. For example, the lack of a benchmark model of vision makes it difficult to test the significance of internal representations against visuo-motor interactions (Figure 5b). This difficulty can make the existence and importance of internal representations inconclusive. Such inconclusiveness, in turn, may help to explain the vigorous and unsettled debates over the nature of this and other speculative features in systems neuroscience (Langdon et al., 2023; Sohal, 2016; Destexhe and Touboul, 2021). By contrast, well-defined benchmark models of cortical networks make it relatively easy to show the redundancy of cores or clubs against these models (Figure 5c). This ease may help explain the lack of comparable debates over the nature of these and other redundant features in network neuroscience (Liao et al., 2017; Sporns, 2018; Pasqualetti et al., 2019).

Suggestive narratives: Just-so stories

Just-so stories in biology are intriguing but speculative narratives that suggest the presence of theoretically elegant or optimal biological function (Gould, 1978; Bowers and Davis, 2012). A just-so-story may suggest, for example, that heart sounds exist to warn of overexertion and thus help minimize energy expenditure. Just-so stories can be difficult to falsify because it is often easy to reexplain some evident non-optimality as a globally optimal tradeoff between competing objectives (Gould and Lewontin, 1979). Table 3 shows examples of such stories in the recent systems-neuroscience literature.

Table 3

Concept	Initial narrative of optimality	Evidence of suboptimality (strong but unviable null model)	Restoration of optimality through the inclusion of an ad hoc tradeoff	Alternative benchmark narrative (strong and viable null model)
Criticality (Fontenele et al., 2019; Wilting and Priesemann, 2019; Nanda et al., 2023)	Brain activity always and exactly balances between order and disorder. This allows it to optimize information transmission and storage.	Brain activity does not always or exactly balance between order and disorder.	Brain activity optimizes the tradeoffs between the benefits of criticality and the competing benefits of flexibility or stability.	Brain activity avoids the extremes of overinhibition and overexcitation and is not optimal over and above this avoidance-of-extremes baseline.
Predictive coding (Sun and Firestone, 2020; Van de Cruys et al., 2020; Seth et al., 2020; Cao, 2020)	Brain activity aims to optimally predict incoming sensory input.	Brain activity optimally predicts sensory input in dark and quiet spaces. Despite this, animals tend not to seek out such spaces.	Brain activity aims to optimize the tradeoffs between predictions that are accurate and predictions that are motivational.	Brain activity reacts to sensory input but does not aim to optimally predict this input.
Wiring minimization (Markov et al., 2013; Bullmore and Sporns, 2012; Rubinov, 2016)	Brain-network structure globally minimizes wiring cost and therefore optimizes wiring economy.	Brain-network structure does not globally minimize wiring cost.	Brain-network structure optimizes the tradeoffs between wiring cost and communication efficiency.	Brain networks have long connections that enable specific sensory-motor function but do not optimize global communication.

The difficulty of falsifying just-so stories also helps to show the value of tests against benchmark models. Assertions of suboptimality form strong but unviable null models (Table 3, third column). Acceptance of these models, in other words, does not offer a viable alternative explanation to replace the original narrative. Without such a viable alternative, it becomes easy to hold on to the original narrative, typically by introducing an ad hoc tradeoff that restores optimality (Table 3, fourth column). This process may help to explain why just-so stories can hold sway in the field long after they are rejected against strong null models. By contrast, benchmark models form strong and viable null models (Table 3, fifth column). The acceptance of these models offers viable alternative explanations of brain function and, in this way, makes it easier to eliminate the original narrative (Appendix 2).

Stagnation and progress

The commonness of circular analyses of knowledge can help explain a seeming disconnect between the fast pace of everyday discovery and the slow pace of real progress. Cobb, 2020 described the nature of this disconnect in neuroscience:

• “There are now tens of thousands of brain researchers around the world, beavering away in a bewildering range of new subdisciplines […] each with their own questions, methods and approaches. Thousands of research articles relating to brain function appear each year.” Despite this, “[i]n reality, no major conceptual innovation has been made in our overall understanding of how the brain works for over half a century.”

On the one hand, circular analyses of knowledge can enable a fast pace of intriguing, and often replicable, everyday discoveries. On the other hand, the speculative and redundant nature of these discoveries does not lead to revisions of benchmark models and, in this way, results in a lack of real progress. Horgan, 2015 introduced the term “ironic science” to describe the nature of this process:

• “Ironic science [acceptance of intriguing but speculative models] offers points of view, opinions, which are, at best, interesting, which provoke further comment. But it does not converge on the truth [lead to acceptance of truer models]. It cannot achieve empirically verifiable surprises that force scientists to make substantial revisions in their basic description of reality [make substantial revisions to benchmark models].”

Tests against benchmark models can help resolve this disconnect by ultimately linking the value of proposed discovery with revisions of benchmark models. Particle physics provides a good example of these tests in action. This field has the Standard Model, perhaps the most successful benchmark model in all of science today. The field seeks to revise this model but refreshingly accepts, and indeed embraces, the everyday failure to do so. Cousins, 2017 aptly summarized the nature of this practice:

• “In many searches in [particle physics], there is a hope to reject the [Standard Model] and make a major discovery […]. But there is nonetheless high (or certainly non-negligible) prior belief in the null hypothesis. The literature, including the most prestigious journals, has many papers […] that report no significant evidence for the sought-for [beyond-the-Standard-Model] physics. Often these publications provide useful constraints on theoretical speculation, and offer guidance for future searches.”

In contrast to particle physics, benchmark models are often ill-defined in more expansive fields, such as psychology or sociology. The difficulty of evaluating real progress in these fields can make practitioners throw up their hands in despair (Yarkoni, 2020 gives an example from psychology). It may also make them avoid tests against null models altogether. For example, Gelman et al., 2020 noted:

• “We do not generally use null hypothesis significance testing in our own work. In the fields in which we work [social science and public health], we do not generally think null hypotheses can be true [cf. strawman models can be truer than speculative models]. We do not find it particularly helpful to formulate and test null hypotheses that we know ahead of time cannot be true [cf. almost invariably accept speculative models against strawman models].”

Systems neuroscience probably lies somewhere between particle physics and social science. Some parts of the field, such as network neuroscience, are sufficiently circumscribed to allow tests of new models against well-delineated benchmark models. To be clear, it is unlikely that the field can converge on benchmark models that resemble the Standard Model or even remotely approach the explanatory success of this model. Despite these limitations, the adoption of routine tests against benchmark models can help place the field on a rigorous foundation and in this way facilitate real progress.

Practical details

This section describes the practical details of testing new models against benchmark models. It first describes steps to integrate existing knowledge into benchmark models. It then discusses methods to sample data from benchmark-model distributions. It finally proposes practical steps to establish a culture of rigorous tests.

Integrating knowledge

Benchmark models should include all aspects of important existing knowledge about some phenomenon of interest. The need to include all knowledge reflects not dogma but the objective importance of control for all known confounding explanations. This need parallels the need to control for all aspects of the noise in tests on independent data (Appendix 3) or the need to control for all confounding explanations in randomized controlled trials (Box 2).

In principle, the inclusion of all important existing knowledge can seem daunting. In practice, however, this inclusion already happens routinely, albeit often informally, in books, reviews, and detailed Introduction sections of original articles. For example, an Introduction section that describes the importance of features a, b, and c to some phenomenon of interest, informally includes all these features in a benchmark model of this phenomenon.

Features that comprise important existing knowledge should rest on rigorous evidence from extensive observations or controlled experiments. Such evidence generally points to strong similarities between the nervous system and other body systems, and to strong similarities between the nervous systems of distantly related species. These similarities span functional objectives, structural building blocks, and developmental processes.

We know, for example, that other body systems use effective but often inelegant tricks to solve diverse but always specific problems of survival and reproduction. We also know that nervous systems use similarly effective but inelegant tricks to feed, fight, flee, mate, and solve other diverse but similarly specific problems (Ramachandran, 1985; Marcus, 2009). We also know that the specific details of these tricks are similar in distantly related species (Nieuwenhuys and Puelles, 2016; Tosches, 2017; Cisek, 2019). These similarities include homologies of specific circuits (Sanes and Zipursky, 2010; Borst and Helmstaedter, 2015; Clark and Demb, 2016), systems (Strausfeld and Hirth, 2013; Fiore et al., 2015; Riebli and Reichert, 2016) and developmental processes (Carroll, 1995; Arthur, 2010; Held, 2017) in flies and mice, organisms that diverged about 600 million years ago (Figure 6). The importance, specificity, and conservation of these features make them natural candidates for inclusion in benchmark models (Appendix 5).

Figure 6

Download asset Open asset

Defining models

Models of the phenomena. Benchmark models of relatively simple or circumscribed phenomena can sometimes take the form of parametric equations. In neuroscience, perhaps the best-known example of such a benchmark model is the Hodgkin-Huxley model of the action potential (Hodgkin and Huxley, 1952). By contrast, benchmark models of complex or expansive phenomena, such as whole-brain networks, are often hard to express in parametric form. These models can instead be defined pragmatically on the basis of benchmark features in empirical data (Table 4). Such data-driven definitions can resemble dimensionality reduction (Cunningham and Ghahramani, 2015) and force studies to formalize the often-vague theoretical concepts as quantifiable model features.

Table 4

Model feature	Example statistic
Sensory-motor interactions	Connectivity and activity statistics of functional circuits.
Excitation/inhibition balance	1/f power-spectral slopes (Gao et al., 2017).
Node connectivity	Degree-distribution statistics (Clauset et al., 2009).
Network clusters	Within-module densities (Fortunato, 2010).
Tuning representations	Tuning-curve statistics (Kriegeskorte and Wei, 2021).
Manifold representations	Persistent-homology barcodes (Ghrist, 2008).
Oscillations	Frequency-specific amplitudes and phases (Donoghue et al., 2020).
Criticality	Avalanche exponents (Sethna et al., 2001).
Small worlds	Small-world statistics (Bassett and Bullmore, 2017).
Cores/clubs	Within-core densities (Csermely et al., 2013).
Network controllability	Network-controllability statistics (Pasqualetti et al., 2014).

Many applied or clinical fields seek to explain the nature of altered brain development, structure, or function. Formulation of benchmark models is equally important in these fields. Benchmark models of altered phenomena should correspondingly be defined in terms of altered, rather than absolute, values of empirical features. For example, benchmark models of neuropsychiatric disorders could be defined in terms of altered development and structure that coherently delineate specific patient populations (Insel and Cuthbert, 2015; Hampel et al., 2023).

Models of the data. In practice, benchmark models should also include features that represent data limitations or biases. For example, limitations of neural-activity data may include acquisition artifacts, physiological confounders and indirectness of neural-activity markers (Hillman, 2014; Wei et al., 2020). The inclusion of these data features in benchmark models can help to mitigate their confounding effects. The interactions of these features with other aspects of the signal, however, makes it ultimately impossible to fully eliminate these effects (Appendix 3).

Sampling data

Tests against benchmark models rest on the ability to sample data from benchmark-model distributions. This sampling should ideally be unbiased: the data samples should match the model statistics but be maximally random otherwise. Unbiased sampling allows us to make valid statistical inferences. For example, the opinions of an unbiased sample of people allow us to make valid statistical inferences about the opinions of the whole population.

In practice, fully unbiased sampling is often intractable, but approximately unbiased sampling is often possible for many interesting benchmark-model distributions. For clarity, this section distinguishes between specific and general methods for doing such sampling.

Specific sampling methods typically first express benchmark-model distributions as solution spaces of data that satisfy benchmark statistics (Schellenberger and Palsson, 2009). They then randomly draw data samples from these solution spaces. Important examples of these methods can sample data with spatial and temporal correlations (Prichard and Theiler, 1994; Roberts et al., 2016; Nanda et al., 2023). The main strength of these methods is in the ability to sample data in fast and unbiased ways. Their main weakness is the inability to sample data with general or arbitrary features and their consequent restriction to a relatively narrow set of benchmark models.

General sampling methods have a complementary set of strengths and weaknesses. The main strength of these methods is the ability to sample data with general or arbitrary benchmark features. Their main weakness is the slow or biased nature of the sampling.

General sampling methods comprise two broad types. The first type of general sampling typically begins with an initial data sample that typically matches the dimensionality and other basic properties of empirical data. It then iteratively randomizes this initial sample in a way that satisfies the benchmark statistics of empirical data, usually by minimizing an error function (Schreiber, 1998). Unbiased sampling requires that this randomization could, in principle, reach all possible samples and that randomization at each iteration could, in principle, be reversible (Newman and Barkema, 1999). These conditions imply that this randomization must be “non-greedy” or not necessarily lower the error at each iteration.

The second type of general sampling typically uses statistical inference methods, such as the principle of maximum entropy. It first defines and fits parametric data distributions and then randomly draws data samples from these distributions (Squartini and Garlaschelli, 2011). In contrast to other sampling methods, this approach preserves the benchmark statistics in the population average but not necessarily in each individual data sample. Fully unbiased sampling with this approach is often intractable for large datasets. Assumptions of independence can make this sampling tractable for many benchmark models but likely at the expense of considerable bias (Cimini et al., 2019).

Making progress

The importance of tests against benchmark models reflects the broader importance of scientific progress. In modern science, the notion of progress is intertwined with the concept of impact. Formally, impact often denotes the number of papers and citations. Implicitly, impact signals real progress. Circular analyses of knowledge enable speculative and redundant results that can lead to many intriguing, replicable, and highly cited papers. Such papers satisfy the formal meaning of impact even as they fail to make real progress (Lawrence, 2007; Alberts, 2013).

Tests against benchmark models can help to align the formal and intuitive definitions of impact. A narrow perspective on genuine impact could equate impact with direct revisions of benchmark models. A broader and more realistic perspective can also emphasize advances that indirectly facilitate revisions of benchmark models (Table 5).

Table 5

Advance	Nature of impact
Discoveries	Revisions of benchmark models (typically rare).
Null results	Rejections of previously promising speculative models.
Exploratory advances	Formulations of newly promising speculative models.
Conceptual advances	Discoveries of explanatory gaps that enable exploratory advances.
Methodological advances	Improvements in data or analysis that support all the other advances.

Separately, the adoption of benchmarking best practices from predictive modeling fields, including machine learning (Weber et al., 2019; Mangul et al., 2019; Mitchell et al., 2019; Kapoor and Narayanan, 2023), can help facilitate progress in explanatory modeling. The following list describes three important examples of these practices:

1. High-quality and publicly accessible data can advance discovery in several ways. First, such data can serve as a reference for formulating consensus benchmark models. Second, such data can help reveal explanatory gaps in existing benchmark models. Third, such data can help formulate new and promising speculative models.

2. Standardized summaries of models and tests can help replace imprecise narratives with quantitative summaries of individual results. Machine-readable versions of these summaries can help facilitate automated integration of such results across studies.

3. A centralized integration of results can help to formalize discovery through continuous revisions of benchmark models. It can also help to collate and standardize null results and, in this way, eliminate rejected speculative models from future tests.

Together, this change in focus can help motivate systems neuroscientists to carefully formulate new models and to rigorously test these models against benchmark models. Such testing can lead to a welcome decrease in publications of speculative and redundant results. And collectively, the resulting alignment of formal and intuitive definitions of impact can give the field a better chance to make real progress.

Concluding recommendations

Circular analyses of noise, and the resulting problem of irreplicable results, form a known impediment to progress in systems neuroscience. This work described that circular analyses of knowledge, and the resulting problem of redundant results, form a less-well known but similarly serious impediment. This concluding section summarizes my overall suggestions for resolving this problem. Appendix 6 discusses objections to some of these suggestions.

Raise awareness. Few scientists and funding bodies formally discuss the problem of redundant results. The lack of this discussion contrasts with extensive parallel discussions of the problem of irreplicable results. The start of this discussion, including in research and policy papers, will be an important first step towards the development of principled solutions.

Reevaluate discoveries. Systematic community efforts should establish the genuine novelty of discoveries in systems and network neuroscience. These efforts may benefit from the experience of similar efforts to establish the replicability of discoveries in psychological and social science (Open Science Collaboration, 2015; Camerer et al., 2018). These efforts face specific challenges, however, including establishing consensus on definitions of reference datasets, benchmark models, and test statistics.

Delimit speculation. Speculation often helps to formulate promising new models. At the same time, misuse of speculation can lead to the neglect of rigorous tests and to the inappropriate acceptance of speculative models. Studies should minimize this misuse by delimiting all suggestively speculative terms (deepities), structure (spandrels), and narratives (just-so stories). Ideally, these delimitations should be prominently made in Introduction sections.

Define benchmarks. Many parts of systems and network neuroscience lack benchmark models. The field should formulate such models to integrate all important existing knowledge and rigorously test proposed discovery. Challenges in the formulation of benchmark models include collation and curation of existing knowledge, consensus definition of model features and test statistics, and development of distinct models for individual phenomena.

Advance sampling. A dearth of powerful sampling methods limits the adoption of rigorous model tests. The field should develop unbiased and scalable methods for sampling data from diverse benchmark-model distributions. Challenges in the development of these methods include competing demands of unbiased sampling and scalability (for general methods) and extensions to diverse benchmark-model distributions (for specific methods).

Reclaim impact. The divergence of formal and intuitive meanings of impact can hinder scientific progress. A multifaceted assessment of direct or indirect impact that centers on revisions of benchmark models, and that discourages redundant explanations, can help to reduce this divergence. Research and funding bodies can emphasize this multifaceted assessment and downplay the use of publication metrics as indicators of progress.

This section provides two complementary perspectives on the concept of relative trueness.

Statistical perspective

This work focuses on scientific analyses that estimate parameters of explanatory models (Appendix 1—figure 1, top). Many such analyses cannot express explanatory models as parametric equations and cannot perform formal parameter estimation. Instead, these analyses often use significance tests to informally estimate parameters of underlying explanatory models.

This work assumes that all parameter estimates are directly comparable across all models. It makes this assumption without loss of generality because any two models can be nested within a more general model. It adopts the terminology of the International Organization for Standardization (ISO, 1994; Menditto et al., 2007) to quantify the accuracy of these parameter estimates. It defines trueness as the inverse of the estimation bias and precision as the inverse of the estimation variance (Appendix 1—figure 1, bottom). Note that trueness has a single and clear meaning, whereas bias is often a catch-all term, especially outside statistics (Danks and London, 2017; Fanelli et al., 2017).

This work focuses on the problem of redundant features in explanatory models. Successful explanatory models tend to have relatively low estimation bias. The inclusion of redundant features tends to increase this bias and thereby reduce explanatory success. By contrast, the work does not consider the problem of redundant features in predictive models. Successful predictive models have relatively low prediction bias but may not necessarily have low estimation bias, as we saw in the above distinction between trueness and utility. The inclusion of redundant features in these models does not necessarily increase their prediction bias (Guyon and Elisseeff, 2003) and, in this way, does not necessarily reduce their predictive success. (Note that this distinction between explanatory and predictive modeling differs from the treatment of Shmueli, 2010).

Appendix 1—figure 1

Download asset Open asset

This section relates circular and unified analyses to Mayo’s framework of severe testing (Mayo and Spanos, 2011; Mayo, 2018). First, it shows that circular analyses of knowledge form a specific violation of Mayo’s weak-severity requirement. Second, it shows that unified analyses form a specific adherence to Mayo’s strong-severity requirement. Third, it describes unified analyses as a type of severe selection: a hybrid approach that combines severe testing with model selection.

This section provides a unified perspective on the circular analyses of noise in Kriegeskorte et al., 2009 and the circular analyses of knowledge in this work. It shows that these two problems, and their corresponding solutions, share deep similarities but also have basic differences. The differences reflect the basic distinctions between overfitting and overspecification.

This section describes a framework for integrating existing knowledge about the function, structure, development, and evolution of individual biological features or traits (Appendix 5—figure 1). A benchmark model of a specific phenomenon should include, where possible, important existing knowledge from all aspects of this framework.

This framework is based on classifications of Mayr and Tinbergen (Mayr, 1961; Tinbergen, 1963; Laland et al., 2011; Bateson and Laland, 2013; Nesse, 2013) and is organized along two dimensions. The first dimension reflects the nature and origin of a trait. It distinguishes what the trait is (structure and function) from how it came to be (development and evolution). The second dimension reflects the mechanistic timescales of this nature and origin. It distinguishes the proximate mechanisms of a single lifetime (structure and development) from the distal mechanisms of many generations (function and evolution).

This framework is widely accepted in biology but less well-known in neuroscience. Instead, more neuroscientists seem to know about Marr’s (and Poggio’s) three-level framework for studying the brain as a computer (Marr, 1982). The first level of this framework (Marr 1) denotes the aim of brain “computation”. The second level (Marr 2) denotes the “algorithms” that achieve this aim. The third level (Marr 3) denotes the “hardware” that implements the algorithms.

The focus on computation alone is somewhat restrictive because it separates the brain from the rest of the body. We have no rigorous evidence to support this separation (see the main text for more discussion). Interestingly, and in line with this observation, Poggio recently updated Marr’s framework to include development and evolution (Poggio, 2012). In this way, he seems to have independently converged on Tinbergen’s more general biological classification.

Appendix 5—figure 1

Download asset Open asset

This section discusses possible objections to the main recommendations in this work.

1. 1. Achard S
   2. Salvador R
   3. Whitcher B
   4. Suckling J
   5. Bullmore E

   (2006) A resilient, low-frequency, small-world human brain functional network with highly connected association cortical hubs

   Journal of Neuroscience 26:63–72.

   https://doi.org/10.1523/JNEUROSCI.3874-05.2006

   • PubMed
   • Google Scholar
2. 1. Aho K
   2. Derryberry D
   3. Peterson T

   (2014) Model selection for ecologists: the worldviews of AIC and BIC

   Ecology 95:631–636.

   https://doi.org/10.1890/13-1452.1

   • PubMed
   • Google Scholar
3. 1. Alberts B

   (2013) Impact factor distortions

   Science 340:787.

   https://doi.org/10.1126/science.1240319

   • PubMed
   • Google Scholar
4. Book

   1. Arthur W

   (2010)

   Evolution: A Developmental Approach

   Wiley.

   • Google Scholar
5. 1. Bailly X
   2. Reichert H
   3. Hartenstein V

   (2013) The urbilaterian brain revisited: novel insights into old questions from new flatworm clades

   Development Genes and Evolution 223:149–157.

   https://doi.org/10.1007/s00427-012-0423-7

   • PubMed
   • Google Scholar
6. 1. Baker A

   (2022)

   Stanford Encyclopedia of Philosophy

   Simplicity, Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University.

   • Google Scholar
7. 1. Baker B
   2. Lansdell B
   3. Kording KP

   (2022) Three aspects of representation in neuroscience

   Trends in Cognitive Sciences 26:942–958.

   https://doi.org/10.1016/j.tics.2022.08.014

   • PubMed
   • Google Scholar
8. Book

   1. Barabási AL

   (2016)

   Network Science

   Cambridge University Press.

   • Google Scholar
9. 1. Bassett DS
   2. Bullmore ET

   (2017) Small-world brain networks revisited

   The Neuroscientist 23:499–516.

   https://doi.org/10.1177/1073858416667720

   • PubMed
   • Google Scholar
10. 1. Bassett DS
    2. Sporns O

    (2017) Network neuroscience

    Nature Neuroscience 20:353–364.

    https://doi.org/10.1038/nn.4502

    • PubMed
    • Google Scholar
11. 1. Bateson P
    2. Laland KN

    (2013) Tinbergen’s four questions: an appreciation and an update

    Trends in Ecology & Evolution 28:712–718.

    https://doi.org/10.1016/j.tree.2013.09.013

    • Google Scholar
12. 1. Bayne T
    2. Brainard D
    3. Byrne RW
    4. Chittka L
    5. Clayton N
    6. Heyes C
    7. Mather J
    8. Ölveczky B
    9. Shadlen M
    10. Suddendorf T
    11. Webb B

    (2019) What is cognition?

    Current Biology 29:R608–R615.

    https://doi.org/10.1016/j.cub.2019.05.044

    • PubMed
    • Google Scholar
13. 1. Bedau MA

    (1997) Weak emergence

    Noûs 31:375–399.

    https://doi.org/10.1111/0029-4624.31.s11.17

    • Google Scholar
14. 1. Bennett MR
    2. Hacker PMS

    (2022)

    Philosophical Foundations of Neuroscience

    79–93, The mereological fallacy in neuroscience, Philosophical Foundations of Neuroscience, Wiley.

    • Google Scholar
15. 1. Borello U
    2. Pierani A

    (2010) Patterning the cerebral cortex: traveling with morphogens

    Current Opinion in Genetics & Development 20:408–415.

    https://doi.org/10.1016/j.gde.2010.05.003

    • Google Scholar
16. 1. Borst A
    2. Helmstaedter M

    (2015) Common circuit design in fly and mammalian motion vision

    Nature Neuroscience 18:1067–1076.

    https://doi.org/10.1038/nn.4050

    • PubMed
    • Google Scholar
17. 1. Bourget D
    2. Chalmers DJ

    (2014) What do philosophers believe?

    Philosophical Studies 170:465–500.

    https://doi.org/10.1007/s11098-013-0259-7

    • Google Scholar
18. 1. Bowers JS
    2. Davis CJ

    (2012) Bayesian just-so stories in psychology and neuroscience

    Psychological Bulletin 138:389–414.

    https://doi.org/10.1037/a0026450

    • PubMed
    • Google Scholar
19. 1. Bowers JS
    2. Malhotra G
    3. Dujmović M
    4. Montero ML
    5. Tsvetkov C
    6. Biscione V
    7. Puebla G
    8. Adolfi F
    9. Hummel JE
    10. Heaton RF
    11. Evans BD
    12. Mitchell J
    13. Blything R

    (2022) Deep problems with neural network models of human vision

    Behavioral and Brain Sciences 2022:1–74.

    https://doi.org/10.1017/S0140525X22002813

    • PubMed
    • Google Scholar
20. 1. Brette R

    (2019) Is coding a relevant metaphor for the brain?

    Behavioral and Brain Sciences 42:e21.

    https://doi.org/10.1017/S0140525X19000049

    • Google Scholar
21. 1. Brunton SL
    2. Kutz JN

    (2019)

    Data-Driven Science and Engineering

    287–291, Controllability and observability, Data-Driven Science and Engineering, Cambridge University Press.

    • Google Scholar
22. 1. Buckner RL
    2. Krienen FM

    (2013) The evolution of distributed association networks in the human brain

    Trends in Cognitive Sciences 17:648–665.

    https://doi.org/10.1016/j.tics.2013.09.017

    • Google Scholar
23. 1. Bullmore E
    2. Sporns O

    (2012) The economy of brain network organization

    Nature Reviews Neuroscience 13:336–349.

    https://doi.org/10.1038/nrn3214

    • PubMed
    • Google Scholar
24. 1. Burwick T

    (2014) The binding problem

    Wiley Interdisciplinary Reviews: Cognitive Science 5:305–315.

    https://doi.org/10.1002/wcs.1279

    • PubMed
    • Google Scholar
25. 1. Buzsáki G
    2. Draguhn A

    (2004) Neuronal oscillations in cortical networks

    Science 304:1926–1929.

    https://doi.org/10.1126/science.1099745

    • PubMed
    • Google Scholar
26. 1. Buzsáki G
    2. Mizuseki K

    (2014) The log-dynamic brain: how skewed distributions affect network operations

    Nature Reviews Neuroscience 15:264–278.

    https://doi.org/10.1038/nrn3687

    • PubMed
    • Google Scholar
27. 1. Camerer CF
    2. Dreber A
    3. Holzmeister F
    4. Ho T-H
    5. Huber J
    6. Johannesson M
    7. Kirchler M
    8. Nave G
    9. Nosek BA
    10. Pfeiffer T
    11. Altmejd A
    12. Buttrick N
    13. Chan T
    14. Chen Y
    15. Forsell E
    16. Gampa A
    17. Heikensten E
    18. Hummer L
    19. Imai T
    20. Isaksson S
    21. Manfredi D
    22. Rose J
    23. Wagenmakers E-J
    24. Wu H

    (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

    Nature Human Behaviour 2:637–644.

    https://doi.org/10.1038/s41562-018-0399-z

    • PubMed
    • Google Scholar
28. 1. Cao R

    (2020) New labels for old ideas: Predictive processing and the interpretation of neural signals

    Review of Philosophy and Psychology 11:517–546.

    https://doi.org/10.1007/s13164-020-00481-x

    • Google Scholar
29. 1. Carroll SB

    (1995) Homeotic genes and the evolution of arthropods and chordates

    Nature 376:479–485.

    https://doi.org/10.1038/376479a0

    • PubMed
    • Google Scholar
30. 1. Chakravartty A

    (2017)

    Stanford Encyclopedia of Philosophy

    Scientific realism, Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University.

    • Google Scholar
31. 1. Churchland PS
    2. Sejnowski TJ

    (2016)

    The Computational Brain

    61–140, Computational overview, The Computational Brain, MIT Press, 10.7551/mitpress/11207.003.0006.

    • Google Scholar
32. 1. Cichy RM
    2. Kaiser D

    (2019) Deep neural networks as scientific models

    Trends in Cognitive Sciences 23:305–317.

    https://doi.org/10.1016/j.tics.2019.01.009

    • Google Scholar
33. 1. Cimini G
    2. Squartini T
    3. Saracco F
    4. Garlaschelli D
    5. Gabrielli A
    6. Caldarelli G

    (2019) The statistical physics of real-world networks

    Nature Reviews Physics 1:58–71.

    https://doi.org/10.1038/s42254-018-0002-6

    • Google Scholar
34. 1. Cisek P

    (2019) Resynthesizing behavior through phylogenetic refinement

    Attention, Perception & Psychophysics 81:2265–2287.

    https://doi.org/10.3758/s13414-019-01760-1

    • PubMed
    • Google Scholar
35. 1. Clark DA
    2. Demb JB

    (2016) Parallel computations in insect and mammalian visual motion processing

    Current Biology 26:R1062–R1072.

    https://doi.org/10.1016/j.cub.2016.08.003

    • Google Scholar
36. 1. Clauset A
    2. Shalizi CR
    3. Newman MEJ

    (2009) Power-law distributions in empirical data

    SIAM Review 51:661–703.

    https://doi.org/10.1137/070710111

    • Google Scholar
37. Book

    1. Cobb M

    (2020)

    The Idea of the Brain: The Past and Future of Neuroscience

    Basic Books.

    • Google Scholar
38. 1. Colizza V
    2. Flammini A
    3. Serrano MA
    4. Vespignani A

    (2006) Detecting rich-club ordering in complex networks

    Nature Physics 2:110–115.

    https://doi.org/10.1038/nphys209

    • Google Scholar
39. 1. Cousins RD

    (2017) The Jeffreys–Lindley paradox and discovery criteria in high energy physics

    Synthese 194:395–432.

    https://doi.org/10.1007/s11229-014-0525-z

    • Google Scholar
40. 1. Craik KJW

    (1943)

    The Nature of Explanation

    50–61, Hypothesis on the nature of thought, The Nature of Explanation, Cambridge University Press.

    • Google Scholar
41. 1. Csermely P
    2. London A
    3. Wu LY
    4. Uzzi B

    (2013) Structure and dynamics of core/periphery networks

    Journal of Complex Networks 1:93–123.

    https://doi.org/10.1093/comnet/cnt016

    • Google Scholar
42. 1. Cunningham JP
    2. Ghahramani Z

    (2015)

    Linear dimensionality reduction: Survey, insights, and generalizations

    Journal of Machine Learning Research 16:2859–2900.

    • Google Scholar
43. Conference

    1. Danks D
    2. London AJ

    (2017) Algorithmic Bias in Autonomous Systems

    Twenty-Sixth International Joint Conference on Artificial Intelligence. pp. 4691–4697.

    https://doi.org/10.24963/ijcai.2017/654

    • Google Scholar
44. 1. Dennett DC

    (2013)

    Intuition Pumps and Other Tools for Thinking

    56–57, What is a Deepity?, Intuition Pumps and Other Tools for Thinking, New York, United States, W. W. Norton.

    • Google Scholar
45. 1. Destexhe A
    2. Touboul JD

    (2021) Is there sufficient evidence for criticality in cortical systems?

    eNeuro 8:ENEURO.0551-20.2021.

    https://doi.org/10.1523/ENEURO.0551-20.2021

    • PubMed
    • Google Scholar
46. 1. Donoghue T
    2. Haller M
    3. Peterson EJ
    4. Varma P
    5. Sebastian P
    6. Gao R
    7. Noto T
    8. Lara AH
    9. Wallis JD
    10. Knight RT
    11. Shestyuk A
    12. Voytek B

    (2020) Parameterizing neural power spectra into periodic and aperiodic components

    Nature Neuroscience 23:1655–1665.

    https://doi.org/10.1038/s41593-020-00744-x

    • PubMed
    • Google Scholar
47. 1. Donoghue T
    2. Schaworonkow N
    3. Voytek B

    (2022) Methodological considerations for studying neural oscillations

    The European Journal of Neuroscience 55:3502–3527.

    https://doi.org/10.1111/ejn.15361

    • PubMed
    • Google Scholar
48. 1. Driscoll LN
    2. Duncker L
    3. Harvey CD

    (2022) Representational drift: Emerging theories for continual learning and experimental future directions

    Current Opinion in Neurobiology 76:102609.

    https://doi.org/10.1016/j.conb.2022.102609

    • PubMed
    • Google Scholar
49. 1. Elsayed GF
    2. Cunningham JP

    (2017) Structure in neural population recordings: an expected byproduct of simpler phenomena?

    Nature Neuroscience 20:1310–1318.

    https://doi.org/10.1038/nn.4617

    • PubMed
    • Google Scholar
50. 1. Fanelli D
    2. Costas R
    3. Ioannidis JPA

    (2017) Meta-assessment of bias in science

    PNAS 114:3714–3719.

    https://doi.org/10.1073/pnas.1618569114

    • PubMed
    • Google Scholar
51. 1. Fiore VG
    2. Dolan RJ
    3. Strausfeld NJ
    4. Hirth F

    (2015) Evolutionarily conserved mechanisms for the selection and maintenance of behavioural activity

    Philosophical Transactions of the Royal Society B 370:20150053.

    https://doi.org/10.1098/rstb.2015.0053

    • Google Scholar
52. 1. Fontenele AJ
    2. de Vasconcelos NAP
    3. Feliciano T
    4. Aguiar LAA
    5. Soares-Cunha C
    6. Coimbra B
    7. Dalla Porta L
    8. Ribeiro S
    9. Rodrigues AJ
    10. Sousa N
    11. Carelli PV
    12. Copelli M

    (2019) Criticality between Cortical States

    Physical Review Letters 122:208101.

    https://doi.org/10.1103/PhysRevLett.122.208101

    • PubMed
    • Google Scholar
53. 1. Fortunato S

    (2010) Community detection in graphs

    Physics Reports 486:75–174.

    https://doi.org/10.1016/j.physrep.2009.11.002

    • Google Scholar
54. Book

    1. Freeman WJ
    2. Skarda CA

    (1990) Representations: who needs them?

    In: Lynch G, McGaugh JL, Weinberger NM, editors. Brain Organization and Memory. Oxford University Press. pp. 375–380.

    https://doi.org/10.1093/oso/9780195077124.003.0022

    • Google Scholar
55. 1. Frégnac Y

    (2017) Big data and the industrialization of neuroscience: A safe roadmap for understanding the brain?

    Science 358:470–477.

    https://doi.org/10.1126/science.aan8866

    • PubMed
    • Google Scholar
56. 1. Frigg R
    2. Hartmann S

    (2020)

    Stanford Encyclopedia of Philosophy

    Models in science, Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University.

    • Google Scholar
57. 1. Gao R
    2. Peterson EJ
    3. Voytek B

    (2017) Inferring synaptic excitation/inhibition balance from field potentials

    NeuroImage 158:70–78.

    https://doi.org/10.1016/j.neuroimage.2017.06.078

    • Google Scholar
58. 1. Geirhos R
    2. Jacobsen JH
    3. Michaelis C
    4. Zemel R
    5. Brendel W
    6. Bethge M
    7. Wichmann FA

    (2020) Shortcut learning in deep neural networks

    Nature Machine Intelligence 2:665–673.

    https://doi.org/10.1038/s42256-020-00257-z

    • Google Scholar
59. 1. Gelman A
    2. Hill J
    3. Vehtari A

    (2020)

    Regression and Other Stories

    49–68, Statistical inference, Regression and Other Stories, Cambridge University Press, 10.1017/9781139161879.

    • Google Scholar
60. 1. Ghrist R

    (2008) Barcodes: The persistent topology of data

    Bulletin of the American Mathematical Society 45:61–76.

    https://doi.org/10.1090/S0273-0979-07-01191-3

    • Google Scholar
61. Preprint

    1. Golan T
    2. Taylor J
    3. Schütt HH
    4. Peters B
    5. Sommers RP
    6. Seeliger K
    7. Doerig A
    8. Linton P
    9. Konkle T
    10. van Gerven M
    11. Kording K
    12. Richards B
    13. Kietzmann TC
    14. Lindsay GW
    15. Kriegeskorte N

    (2023) Deep Neural Networks Are Not a Single Hypothesis but a Language for Expressing Computational Hypotheses

    PsyArXiv.

    https://doi.org/10.31234/osf.io/tr7gx

    • Google Scholar
62. 1. Goodale MA

    (1996)

    Visuomotor modules in the vertebrate brain

    Canadian Journal of Physiology and Pharmacology 74:390–400.

    • PubMed
    • Google Scholar
63. 1. Gould SJ

    (1978)

    Sociobiology: the art of storytelling

    New Scientist 80:530–533.

    • PubMed
    • Google Scholar
64. 1. Gould SJ
    2. Lewontin RC

    (1979) The spandrels of San Marco and the Panglossian paradigm: a critique of the adaptationist programme

    Proceedings of the Royal Society of London. Series B. Biological Sciences 205:581–598.

    https://doi.org/10.1098/rspb.1979.0086

    • Google Scholar
65. 1. Gray CM
    2. König P
    3. Engel AK
    4. Singer W

    (1989) Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties

    Nature 338:334–337.

    https://doi.org/10.1038/338334a0

    • PubMed
    • Google Scholar
66. 1. Griffa A
    2. van den Heuvel MP

    (2018) Rich-club neurocircuitry: function, evolution, and vulnerability

    Dialogues in Clinical Neuroscience 20:121–132.

    https://doi.org/10.31887/DCNS.2018.20.2/agriffa

    • PubMed
    • Google Scholar
67. Book

    1. Grove EA
    2. Monuki ES

    (2020) Morphogens, Patterning centers, and their mechanisms of action

    In: Rubenstein J, Rakic P, Chen B, Kwan KY, editors. Patterning and Cell Type Specification in the Developing CNS and PNS. 2nd Ed. Academic Press. pp. 3–21.

    https://doi.org/10.1016/B978-0-12-814405-3.00001-1

    • Google Scholar
68. 1. Gu S
    2. Pasqualetti F
    3. Cieslak M
    4. Telesford QK
    5. Yu AB
    6. Kahn AE
    7. Medaglia JD
    8. Vettel JM
    9. Miller MB
    10. Grafton ST
    11. Bassett DS

    (2015) Controllability of structural brain networks

    Nature Communications 6:8414.

    https://doi.org/10.1038/ncomms9414

    • PubMed
    • Google Scholar
69. 1. Guyon I
    2. Elisseeff A

    (2003)

    An introduction to variable and feature selection

    Journal of Machine Learning Research 3:1157–1182.

    • Google Scholar
70. 1. Hagmann P
    2. Cammoun L
    3. Gigandet X
    4. Meuli R
    5. Honey CJ
    6. Wedeen VJ
    7. Sporns O

    (2008) Mapping the structural core of human cerebral cortex

    PLOS Biology 6:e159.

    https://doi.org/10.1371/journal.pbio.0060159

    • PubMed
    • Google Scholar
71. 1. Hampel H
    2. Gao P
    3. Cummings J
    4. Toschi N
    5. Thompson PM
    6. Hu Y
    7. Cho M
    8. Vergallo A

    (2023) The foundation and architecture of precision medicine in neurology and psychiatry

    Trends in Neurosciences 46:176–198.

    https://doi.org/10.1016/j.tins.2022.12.004

    • PubMed
    • Google Scholar
72. Book

    1. Held LI

    (2017) Deep Homology? Uncanny Similarities of Humans and Flies Uncovered by Evo-Devo

    Cambridge University Press.

    https://doi.org/10.1017/9781316550175

    • Google Scholar
73. 1. Hermes D
    2. Miller KJ
    3. Wandell BA
    4. Winawer J

    (2015a) Gamma oscillations in visual cortex: the stimulus matters

    Trends in Cognitive Sciences 19:57–58.

    https://doi.org/10.1016/j.tics.2014.12.009

    • Google Scholar
74. 1. Hermes D
    2. Miller KJ
    3. Wandell BA
    4. Winawer J

    (2015b) Stimulus dependence of gamma oscillations in human visual cortex

    Cerebral Cortex 25:2951–2959.

    https://doi.org/10.1093/cercor/bhu091

    • PubMed
    • Google Scholar
75. 1. Hillman EMC

    (2014) Coupling mechanism and significance of the BOLD signal: A status report

    Annual Review of Neuroscience 37:161–181.

    https://doi.org/10.1146/annurev-neuro-071013-014111

    • PubMed
    • Google Scholar
76. 1. Hodgkin AL
    2. Huxley AF

    (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve

    Journal of Physiology 117:500–544.

    https://doi.org/10.1113/jphysiol.1952.sp004764

    • PubMed
    • Google Scholar
77. Book

    1. Horgan J

    (2015)

    The End Of Science: Facing The Limits Of Knowledge In The Twilight Of The Scientific Age

    Basic Books.

    • Google Scholar
78. 1. Hubel DH
    2. Wiesel TN

    (1959) Receptive fields of single neurones in the cat’s striate cortex

    Journal of Physiology 148:574–591.

    https://doi.org/10.1113/jphysiol.1959.sp006308

    • Google Scholar
79. 1. Insel TR
    2. Cuthbert BN

    (2015) Medicine: brain disorders? Precisely

    Science 348:499–500.

    https://doi.org/10.1126/science.aab2358

    • PubMed
    • Google Scholar
80. 1. Ioannidis JPA

    (2005) Why most published research findings are false

    PLOS Medicine 2:e124.

    https://doi.org/10.1371/journal.pmed.0020124

    • PubMed
    • Google Scholar
81. 1. Isaacson JS
    2. Scanziani M

    (2011) How inhibition shapes cortical activity

    Neuron 72:231–243.

    https://doi.org/10.1016/j.neuron.2011.09.027

    • PubMed
    • Google Scholar
82. Report

    1. ISO

    (1994)

    Accuracy (trueness and precision) of measurement methods and results

    ISO.

    • Google Scholar
83. 1. Jonas E
    2. Kording KP

    (2017) Could a neuroscientist understand a microprocessor?

    PLOS Computational Biology 13:e1005268.

    https://doi.org/10.1371/journal.pcbi.1005268

    • PubMed
    • Google Scholar
84. Book

    1. Kaas JH

    (1995) The segregation of function in the nervous system: Why do sensory systems have so many subdivisions

    In: Neff WD, editors. Contributions to Sensory Physiology. Elsevier. pp. 201–240.

    https://doi.org/10.1016/B978-0-12-151807-3.50012-4

    • Google Scholar
85. 1. Kaiser M
    2. Hilgetag CC

    (2006) Nonoptimal component placement, but short processing paths, due to long-distance projections in neural systems

    PLOS Computational Biology 2:e95.

    https://doi.org/10.1371/journal.pcbi.0020095

    • PubMed
    • Google Scholar
86. 1. Kapoor S
    2. Narayanan A

    (2023) Leakage and the reproducibility crisis in machine-learning-based science

    Patterns 4:100804.

    https://doi.org/10.1016/j.patter.2023.100804

    • Google Scholar
87. Book

    1. Kenny A

    (1971)

    The homunculus fallacy

    In: Grene MG, Prigogine I, editors. Interpretations of Life and Mind. Humanities Press. pp. 65–74.

    • Google Scholar
88. 1. Krakauer JW
    2. Ghazanfar AA
    3. Gomez-Marin A
    4. MacIver MA
    5. Poeppel D

    (2017) Neuroscience needs behavior: Correcting a reductionist bias

    Neuron 93:480–490.

    https://doi.org/10.1016/j.neuron.2016.12.041

    • Google Scholar
89. 1. Kriegeskorte N
    2. Simmons WK
    3. Bellgowan PSF
    4. Baker CI

    (2009) Circular analysis in systems neuroscience: the dangers of double dipping

    Nature Neuroscience 12:535–540.

    https://doi.org/10.1038/nn.2303

    • PubMed
    • Google Scholar
90. 1. Kriegeskorte N

    (2015) Deep neural networks: A new framework for modeling biological vision and brain information processing

    Annual Review of Vision Science 1:417–446.

    https://doi.org/10.1146/annurev-vision-082114-035447

    • PubMed
    • Google Scholar
91. 1. Kriegeskorte N
    2. Wei XX

    (2021) Neural tuning and representational geometry

    Nature Reviews Neuroscience 22:703–718.

    https://doi.org/10.1038/s41583-021-00502-3

    • PubMed
    • Google Scholar
92. Conference

    1. Krizhevsky A
    2. Sutskever I
    3. Hinton GE

    (2012)

    Imagenet classification with deep convolutional neural networks

    Advances in Neural Information Processing Systems. pp. 1097–1105.

    • Google Scholar
93. 1. Krubitzer L

    (1995) The organization of neocortex in mammals: are species differences really so different?

    Trends in Neurosciences 18:408–417.

    https://doi.org/10.1016/0166-2236(95)93938-t

    • PubMed
    • Google Scholar
94. 1. Krubitzer LA
    2. Prescott TJ

    (2018) The combinatorial creature: Cortical phenotypes within and across lifetimes

    Trends in Neurosciences 41:744–762.

    https://doi.org/10.1016/j.tins.2018.08.002

    • PubMed
    • Google Scholar
95. Book

    1. Lakatos I

    (1976) Falsification and the methodology of scientific research programmes

    In: Harding SG, editors. Can Theories Be Refuted? Essays on the Duhem-Quine Thesis. Springer. pp. 205–259.

    https://doi.org/10.1007/978-94-010-1863-0

    • Google Scholar
96. 1. Laland KN
    2. Sterelny K
    3. Odling-Smee J
    4. Hoppitt W
    5. Uller T

    (2011) Cause and effect in biology revisited: is Mayr’s proximate-ultimate dichotomy still useful?

    Science 334:1512–1516.

    https://doi.org/10.1126/science.1210879

    • PubMed
    • Google Scholar
97. 1. Langdon C
    2. Genkin M
    3. Engel TA

    (2023) A unifying perspective on neural manifolds and circuits for cognition

    Nature Reviews Neuroscience 24:363–377.

    https://doi.org/10.1038/s41583-023-00693-x

    • Google Scholar
98. 1. Lawrence PA

    (2007) The mismeasurement of science

    Current Biology 17:R583–R585.

    https://doi.org/10.1016/j.cub.2007.06.014

    • PubMed
    • Google Scholar
99. 1. Liao X
    2. Vasilakos AV
    3. He Y

    (2017) Small-world human brain networks: Perspectives and challenges

    Neuroscience and Biobehavioral Reviews 77:286–300.

    https://doi.org/10.1016/j.neubiorev.2017.03.018

    • PubMed
    • Google Scholar
100. 1. Lipton ZC
     2. Steinhardt J

     (2019) Troubling trends in machine learning scholarship

     Queue 17:45–77.

     https://doi.org/10.1145/3317287.3328534

     • Google Scholar
101. 1. Ma Z
     2. Turrigiano GG
     3. Wessel R
     4. Hengen KB

     (2019) Cortical circuit dynamics are homeostatically tuned to criticality in vivo

     Neuron 104:655–664.

     https://doi.org/10.1016/j.neuron.2019.08.031

     • PubMed
     • Google Scholar
102. 1. Mangul S
     2. Martin LS
     3. Hill BL
     4. Lam AKM
     5. Distler MG
     6. Zelikovsky A
     7. Eskin E
     8. Flint J

     (2019) Systematic benchmarking of omics computational tools

     Nature Communications 10:1393.

     https://doi.org/10.1038/s41467-019-09406-4

     • PubMed
     • Google Scholar
103. Book

     1. Marcus G

     (2009)

     Kluge: The Haphazard Evolution of the Human Mind

     Houghton Mifflin.

     • Google Scholar
104. 1. Markov NT
     2. Ercsey-Ravasz M
     3. Van Essen DC
     4. Knoblauch K
     5. Toroczkai Z
     6. Kennedy H

     (2013) Cortical high-density counterstream architectures

     Science 342:1238406.

     https://doi.org/10.1126/science.1238406

     • PubMed
     • Google Scholar
105. Book

     1. Marr D

     (1982)

     Vision: A Computational Investigation into the Human Representation and Processing of Visual Information

     W.H. Freeman.

     • Google Scholar
106. Book

     1. Mayo DG
     2. Spanos A

     (2011)

     Error Statistics

     In: Bandyopadhyay PS, Forster MR, editors. Handbook of the Philosophy of Science. Amsterdam: North-Holland. pp. 153–198.

     • Google Scholar
107. Book

     1. Mayo DG

     (2018) Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars

     Cambridge University Press.

     https://doi.org/10.1017/9781107286184

     • Google Scholar
108. 1. Mayr E

     (1961) Cause and effect in biology

     Science 134:1501–1506.

     https://doi.org/10.1126/science.134.3489.1501

     • PubMed
     • Google Scholar
109. Book

     1. McCall Smith A

     (2007)

     The Good Husband of Zebra Drive

     Knopf Doubleday Publishing Group.

     • Google Scholar
110. 1. Menditto A
     2. Patriarca M
     3. Magnusson B

     (2007) Understanding the meaning of accuracy, trueness and precision

     Accreditation and Quality Assurance 12:45–47.

     https://doi.org/10.1007/s00769-006-0191-z

     • Google Scholar
111. 1. Merker B
     2. Williford K
     3. Rudrauf D

     (2022) The integrated information theory of consciousness: a case of mistaken identity

     Behavioral and Brain Sciences 45:e41.

     https://doi.org/10.1017/S0140525X21000881

     • Google Scholar
112. 1. Mesulam MM

     (1998) From sensation to cognition

     Brain 121 (Pt 6):1013–1052.

     https://doi.org/10.1093/brain/121.6.1013

     • PubMed
     • Google Scholar
113. 1. Meunier D
     2. Lambiotte R
     3. Bullmore ET

     (2010) Modular and hierarchically modular organization of brain networks

     Frontiers in Neuroscience 4:200.

     https://doi.org/10.3389/fnins.2010.00200

     • PubMed
     • Google Scholar
114. Conference

     1. Mitchell M
     2. Wu S
     3. Zaldivar A
     4. Barnes P
     5. Vasserman L
     6. Hutchinson B
     7. Spitzer E
     8. Raji ID
     9. Gebru T

     (2019) Model cards for model reporting

     FAT* ’19. pp. 220–229.

     https://doi.org/10.1145/3287560.3287596

     • Google Scholar
115. Book

     1. Mizrahi M

     (2020) The Relativity of Theory: Key Positions and Arguments in the Contemporary Scientific Realism/Antirealism Debate

     Springer.

     https://doi.org/10.1007/978-3-030-58047-6

     • Google Scholar
116. 1. Mobbs D
     2. Trimmer PC
     3. Blumstein DT
     4. Dayan P

     (2018) Foraging for foundations in decision neuroscience: insights from ethology

     Nature Reviews Neuroscience 19:419–427.

     https://doi.org/10.1038/s41583-018-0010-7

     • PubMed
     • Google Scholar
117. 1. Musgrave AE

     (1973) Falsification and its critics

     Studies in Logic and the Foundations of Mathematics 74:393–406.

     https://doi.org/10.1016/S0049-237X(09)70374-X

     • Google Scholar
118. 1. Nanda A
     2. Johnson GW
     3. Mu Y
     4. Ahrens MB
     5. Chang C
     6. Englot DJ
     7. Breakspear M
     8. Rubinov M

     (2023) Time-resolved correlation of distributed brain activity tracks E-I balance and accounts for diverse scale-free phenomena

     Cell Reports 42:112254.

     https://doi.org/10.1016/j.celrep.2023.112254

     • PubMed
     • Google Scholar
119. 1. Nesse RM

     (2013) Tinbergen’s four questions, organized: a response to Bateson and Laland

     Trends in Ecology & Evolution 28:681–682.

     https://doi.org/10.1016/j.tree.2013.10.008

     • Google Scholar
120. Book

     1. Newman MEJ
     2. Barkema GT

     (1999) Monte Carlo Methods in Statistical Physics

     Clarendon Press.

     https://doi.org/10.1093/oso/9780198517962.001.0001

     • Google Scholar
121. Book

     1. Nieuwenhuys R
     2. Puelles L

     (2016) Towards a New Neuromorphology

     Springer.

     https://doi.org/10.1007/978-3-319-25693-1

     • Google Scholar
122. 1. Nilsson DE

     (2021) The diversity of eyes and vision

     Annual Review of Vision Science 7:19–41.

     https://doi.org/10.1146/annurev-vision-121820-074736

     • PubMed
     • Google Scholar
123. 1. Nizami L

     (2019)

     Information theory is abused in neuroscience

     Cybernetics & Human Knowing 26:47–97.

     • Google Scholar
124. 1. Oddie G
     2. Cevolani G

     (2022)

     Stanford Encyclopedia of Philosophy

     Truthlikeness, Stanford Encyclopedia of Philosophy, Metaphysics Research Lab, Stanford University.

     • Google Scholar
125. Book

     1. Oerter R

     (2006)

     The Theory of Almost Everything: The Standard Model, the Unsung Triumph of Modern Physics

     Penguin.

     • Google Scholar
126. 1. Open Science Collaboration

     (2015) Estimating the reproducibility of psychological science

     Science 349:6251.

     https://doi.org/10.1126/science.aac4716

     • Google Scholar
127. 1. Orr HA

     (2009) Fitness and its role in evolutionary genetics

     Nature Reviews Genetics 10:531–539.

     https://doi.org/10.1038/nrg2603

     • PubMed
     • Google Scholar
128. 1. Pasqualetti F
     2. Zampieri S
     3. Bullo F

     (2014) Controllability metrics, limitations and algorithms for complex networks

     IEEE Transactions on Control of Network Systems 1:40–52.

     https://doi.org/10.1109/TCNS.2014.2310254

     • Google Scholar
129. 1. Pasqualetti F
     2. Gu S
     3. Bassett DS

     (2019) RE: Warnings and caveats in brain controllability

     NeuroImage 197:586–588.

     https://doi.org/10.1016/j.neuroimage.2019.05.001

     • PubMed
     • Google Scholar
130. 1. Patankar SP
     2. Kim JZ
     3. Pasqualetti F
     4. Bassett DS

     (2020) Path-dependent connectivity, not modularity, consistently predicts controllability of structural brain networks

     Network Neuroscience 4:1091–1121.

     https://doi.org/10.1162/netn_a_00157

     • PubMed
     • Google Scholar
131. 1. Patel GH
     2. Kaplan DM
     3. Snyder LH

     (2014) Topographic organization in the brain: searching for general principles

     Trends in Cognitive Sciences 18:351–363.

     https://doi.org/10.1016/j.tics.2014.03.008

     • Google Scholar
132. 1. Poggio T

     (2012) The Levels of Understanding framework, revised

     Perception 41:1017–1023.

     https://doi.org/10.1068/p7299

     • PubMed
     • Google Scholar
133. 1. Poggio T
     2. Serre T

     (2013) Models of visual cortex

     Scholarpedia 8:3516.

     https://doi.org/10.4249/scholarpedia.3516

     • Google Scholar
134. Book

     1. Popper KR

     (1963)

     Conjectures and Refutations: The Growth of Scientific Knowledge

     Routledge & K. Paul.

     • Google Scholar
135. 1. Prichard D
     2. Theiler J

     (1994) Generating surrogate data for time series with several simultaneously measured variables

     Physical Review Letters 73:951–954.

     https://doi.org/10.1103/PhysRevLett.73.951

     • PubMed
     • Google Scholar
136. 1. Ramachandran VS

     (1985) The neurobiology of perception

     Perception 14:97–103.

     https://doi.org/10.1068/p140097

     • PubMed
     • Google Scholar
137. 1. Ray S
     2. Maunsell JHR

     (2015) Do gamma oscillations play a role in cerebral cortex?

     Trends in Cognitive Sciences 19:78–85.

     https://doi.org/10.1016/j.tics.2014.12.002

     • Google Scholar
138. 1. Reid AT
     2. Headley DB
     3. Mill RD
     4. Sanchez-Romero R
     5. Uddin LQ
     6. Marinazzo D
     7. Lurie DJ
     8. Valdés-Sosa PA
     9. Hanson SJ
     10. Biswal BB
     11. Calhoun V
     12. Poldrack RA
     13. Cole MW

     (2019) Advancing functional connectivity research from association to causation

     Nature Neuroscience 22:1751–1760.

     https://doi.org/10.1038/s41593-019-0510-4

     • PubMed
     • Google Scholar
139. 1. Richards BA
     2. Lillicrap TP
     3. Beaudoin P
     4. Bengio Y
     5. Bogacz R
     6. Christensen A
     7. Clopath C
     8. Costa RP
     9. de Berker A
     10. Ganguli S
     11. Gillon CJ
     12. Hafner D
     13. Kepecs A
     14. Kriegeskorte N
     15. Latham P
     16. Lindsay GW
     17. Miller KD
     18. Naud R
     19. Pack CC
     20. Poirazi P
     21. Roelfsema P
     22. Sacramento J
     23. Saxe A
     24. Scellier B
     25. Schapiro AC
     26. Senn W
     27. Wayne G
     28. Yamins D
     29. Zenke F
     30. Zylberberg J
     31. Therien D
     32. Kording KP

     (2019) A deep learning framework for neuroscience

     Nature Neuroscience 22:1761–1770.

     https://doi.org/10.1038/s41593-019-0520-2

     • PubMed
     • Google Scholar
140. Book

     1. Riebli N
     2. Reichert H

     (2016) The first nervous system

     In: Shepherd SV, editors. The Wiley Handbook of Evolutionary Neuroscience. Wiley. pp. 125–152.

     https://doi.org/10.1002/9781118316757

     • Google Scholar
141. 1. Roberts JA
     2. Perry A
     3. Lord AR
     4. Roberts G
     5. Mitchell PB
     6. Smith RE
     7. Calamante F
     8. Breakspear M

     (2016) The contribution of geometry to the human connectome

     NeuroImage 124:379–393.

     https://doi.org/10.1016/j.neuroimage.2015.09.009

     • PubMed
     • Google Scholar
142. 1. Roux E

     (2014) The concept of function in modern physiology

     Journal of Physiology 592:2245–2249.

     https://doi.org/10.1113/jphysiol.2014.272062

     • Google Scholar
143. 1. Rubinov M

     (2015) Neural networks in the future of neuroscience research

     Nature Reviews Neuroscience 16:767.

     https://doi.org/10.1038/nrn4042

     • PubMed
     • Google Scholar
144. 1. Rubinov M

     (2016) Constraints and spandrels of interareal connectomes

     Nature Communications 7:13812.

     https://doi.org/10.1038/ncomms13812

     • PubMed
     • Google Scholar
145. Software

     1. Rubinov M

     (2023) Pipeline for semi-automated analysis of network-neuroscience literature, version swh:1:rev:dc5a056851b7b6ba4d6aae34447eb759f62db9e1

     Software Heritage.

     https://archive.softwareheritage.org/swh:1:dir:94844f44be5568182054c9c70665c57be12a4810;origin=https://github.com/mikarubi/litrev;visit=swh:1:snp:d42d9bc6260b2448a6fb718496a4f1070e525492;anchor=swh:1:rev:dc5a056851b7b6ba4d6aae34447eb759f62db9e1
146. 1. Sanes JR
     2. Zipursky SL

     (2010) Design principles of insect and vertebrate visual systems

     Neuron 66:15–36.

     https://doi.org/10.1016/j.neuron.2010.01.018

     • PubMed
     • Google Scholar
147. 1. Schellenberger J
     2. Palsson BØ

     (2009) Use of randomized sampling for analysis of metabolic networks

     Journal of Biological Chemistry 284:5457–5461.

     https://doi.org/10.1074/jbc.R800048200

     • PubMed
     • Google Scholar
148. 1. Schreiber T

     (1998) Constrained randomization of time series data

     Physical Review Letters 80:2105–2108.

     https://doi.org/10.1103/PhysRevLett.80.2105

     • Google Scholar
149. 1. Seth AK
     2. Millidge B
     3. Buckley CL
     4. Tschantz A

     (2020) Curious inferences: Reply to Sun and Firestone on the Dark Room Problem

     Trends in Cognitive Sciences 24:681–683.

     https://doi.org/10.1016/j.tics.2020.05.011

     • Google Scholar
150. 1. Sethna JP
     2. Dahmen KA
     3. Myers CR

     (2001) Crackling noise

     Nature 410:242–250.

     https://doi.org/10.1038/35065675

     • PubMed
     • Google Scholar
151. 1. Shackel N

     (2005) The vacuity of postmodernist methodology

     Metaphilosophy 36:295–320.

     https://doi.org/10.1111/j.1467-9973.2005.00370.x

     • Google Scholar
152. 1. Shew WL
     2. Yang H
     3. Petermann T
     4. Roy R
     5. Plenz D

     (2009) Neuronal avalanches imply maximum dynamic range in cortical networks at criticality

     Journal of Neuroscience 29:15595–15600.

     https://doi.org/10.1523/JNEUROSCI.3864-09.2009

     • PubMed
     • Google Scholar
153. 1. Shew WL
     2. Plenz D

     (2013) The functional benefits of criticality in the cortex

     The Neuroscientist 19:88–100.

     https://doi.org/10.1177/1073858412445487

     • PubMed
     • Google Scholar
154. 1. Shmueli G

     (2010) To explain or to predict?

     Statistical Science 25:289–310.

     https://doi.org/10.1214/10-STS330

     • Google Scholar
155. 1. Sibbald B
     2. Roland M

     (1998) Understanding controlled trials: why are randomised controlled trials important?

     BMJ 316:201.

     https://doi.org/10.1136/bmj.316.7126.201

     • PubMed
     • Google Scholar
156. 1. Siddiqi SH
     2. Kording KP
     3. Parvizi J
     4. Fox MD

     (2022) Causal mapping of human brain function

     Nature Reviews Neuroscience 23:361–375.

     https://doi.org/10.1038/s41583-022-00583-8

     • PubMed
     • Google Scholar
157. 1. Singer W

     (2001) Consciousness and the binding problem

     Annals of the New York Academy of Sciences 929:123–146.

     https://doi.org/10.1111/j.1749-6632.2001.tb05712.x

     • PubMed
     • Google Scholar
158. 1. Singer W

     (2018) Neuronal oscillations: unavoidable and useful?

     The European Journal of Neuroscience 48:2389–2398.

     https://doi.org/10.1111/ejn.13796

     • PubMed
     • Google Scholar
159. 1. Sohal VS

     (2016) How close are we to understanding what (if anything) γ oscillations do in cortical circuits?

     Journal of Neuroscience 36:10489–10495.

     https://doi.org/10.1523/JNEUROSCI.0990-16.2016

     • PubMed
     • Google Scholar
160. Book

     1. Sokal A

     (2010)

     Beyond the Hoax: Science, Philosophy and Culture

     Oxford University Press.

     • Google Scholar
161. 1. Sporns O
     2. Zwi JD

     (2004) The small world of the cerebral cortex

     Neuroinformatics 2:145–162.

     https://doi.org/10.1385/NI:2:2:145

     • PubMed
     • Google Scholar
162. 1. Sporns O

     (2013) Network attributes for segregation and integration in the human brain

     Current Opinion in Neurobiology 23:162–171.

     https://doi.org/10.1016/j.conb.2012.11.015

     • PubMed
     • Google Scholar
163. 1. Sporns O

     (2018) Graph theory methods: applications in brain networks

     Dialogues in Clinical Neuroscience 20:111–121.

     https://doi.org/10.31887/DCNS.2018.20.2/osporns

     • PubMed
     • Google Scholar
164. 1. Squartini T
     2. Garlaschelli D

     (2011) Analytical maximum-likelihood method to detect patterns in real networks

     New Journal of Physics 13:083001.

     https://doi.org/10.1088/1367-2630/13/8/083001

     • Google Scholar
165. Book

     1. Stangor C
     2. Walinga J

     (2014)

     Introduction to Psychology

     BCcampus.

     • Google Scholar
166. 1. Stephan KE
     2. Hilgetag CC
     3. Burns GA
     4. O’Neill MA
     5. Young MP
     6. Kötter R

     (2000) Computational analysis of functional connectivity between areas of primate cerebral cortex

     Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 355:111–126.

     https://doi.org/10.1098/rstb.2000.0552

     • PubMed
     • Google Scholar
167. 1. Strausfeld NJ
     2. Hirth F

     (2013) Deep homology of arthropod central complex and vertebrate basal ganglia

     Science 340:157–161.

     https://doi.org/10.1126/science.1231828

     • PubMed
     • Google Scholar
168. 1. Sun Z
     2. Firestone C

     (2020) The Dark Room Problem

     Trends in Cognitive Sciences 24:346–348.

     https://doi.org/10.1016/j.tics.2020.02.006

     • Google Scholar
169. 1. Tang Y
     2. Gao H
     3. Zou W
     4. Kurths J

     (2012) Identifying controlling nodes in neuronal networks in different scales

     PLOS ONE 7:e41375.

     https://doi.org/10.1371/journal.pone.0041375

     • PubMed
     • Google Scholar
170. 1. Tang E
     2. Bassett DS

     (2018) Colloquium: Control of dynamics in brain networks

     Reviews of Modern Physics 90:31003.

     https://doi.org/10.1103/RevModPhys.90.031003

     • Google Scholar
171. 1. Tinbergen N

     (1963) On aims and methods of ethology

     Zeitschrift Für Tierpsychologie 20:410–433.

     https://doi.org/10.1111/j.1439-0310.1963.tb01161.x

     • Google Scholar
172. 1. Tosches MA

     (2017) Developmental and genetic mechanisms of neural circuit evolution

     Developmental Biology 431:16–25.

     https://doi.org/10.1016/j.ydbio.2017.06.016

     • PubMed
     • Google Scholar
173. 1. Tu C
     2. Rocha RP
     3. Corbetta M
     4. Zampieri S
     5. Zorzi M
     6. Suweis S

     (2018) Warnings and caveats in brain controllability

     NeuroImage 176:83–91.

     https://doi.org/10.1016/j.neuroimage.2018.04.010

     • PubMed
     • Google Scholar
174. 1. Van de Cruys S
     2. Friston KJ
     3. Clark A

     (2020) Controlled optimism: Reply to Sun and Firestone on the Dark Room Problem

     Trends in Cognitive Sciences 24:680–681.

     https://doi.org/10.1016/j.tics.2020.05.012

     • Google Scholar
175. 1. van den Heuvel MP
     2. Sporns O

     (2011) Rich-club organization of the human connectome

     Journal of Neuroscience 31:15775–15786.

     https://doi.org/10.1523/JNEUROSCI.3539-11.2011

     • PubMed
     • Google Scholar
176. Book

     1. van Fraassen BC

     (1980) The Scientific Image

     Clarendon Press.

     https://doi.org/10.1093/0198244274.001.0001

     • Google Scholar
177. 1. Vazza F
     2. Feletti A

     (2020) The quantitative comparison between the neuronal network and the cosmic web

     Frontiers in Physics 8:525731.

     https://doi.org/10.3389/fphy.2020.525731

     • Google Scholar
178. 1. Vogelsberger M
     2. Genel S
     3. Springel V
     4. Torrey P
     5. Sijacki D
     6. Xu D
     7. Snyder G
     8. Bird S
     9. Nelson D
     10. Hernquist L

     (2014) Properties of galaxies reproduced by a hydrodynamic simulation

     Nature 509:177–182.

     https://doi.org/10.1038/nature13316

     • PubMed
     • Google Scholar
179. 1. Vul E
     2. Harris C
     3. Winkielman P
     4. Pashler H

     (2009) Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition

     Perspectives on Psychological Science 4:274–290.

     https://doi.org/10.1111/j.1745-6924.2009.01125.x

     • PubMed
     • Google Scholar
180. 1. Wallisch P
     2. Movshon JA

     (2008) Structure and function come unglued in the visual cortex

     Neuron 60:195–197.

     https://doi.org/10.1016/j.neuron.2008.10.008

     • PubMed
     • Google Scholar
181. 1. Wasserstein RL
     2. Lazar NA

     (2016) The ASA statement on p-values: Context, process, and purpose

     The American Statistician 70:129–133.

     https://doi.org/10.1080/00031305.2016.1154108

     • Google Scholar
182. 1. Watts DJ
     2. Strogatz SH

     (1998) Collective dynamics of “small-world” networks

     Nature 393:440–442.

     https://doi.org/10.1038/30918

     • PubMed
     • Google Scholar
183. 1. Weber LM
     2. Saelens W
     3. Cannoodt R
     4. Soneson C
     5. Hapfelmeier A
     6. Gardner PP
     7. Boulesteix AL
     8. Saeys Y
     9. Robinson MD

     (2019) Essential guidelines for computational method benchmarking

     Genome Biology 20:125.

     https://doi.org/10.1186/s13059-019-1738-8

     • PubMed
     • Google Scholar
184. 1. Wei Z
     2. Lin BJ
     3. Chen TW
     4. Daie K
     5. Svoboda K
     6. Druckmann S

     (2020) A comparison of neuronal population dynamics measured with calcium imaging and electrophysiology

     PLOS Computational Biology 16:e1008198.

     https://doi.org/10.1371/journal.pcbi.1008198

     • PubMed
     • Google Scholar
185. 1. Wilting J
     2. Priesemann V

     (2019) 25 years of criticality in neuroscience - established results, open controversies, novel concepts

     Current Opinion in Neurobiology 58:105–111.

     https://doi.org/10.1016/j.conb.2019.08.002

     • PubMed
     • Google Scholar
186. Book

     1. Worrall J

     (1978) The ways in which the methodology of scientific research programmes improves on Popper’s methodology

     In: Radnitzky G, Andersson G, editors. Progress and Rationality in Science. Springer. pp. 45–70.

     https://doi.org/10.1007/978-94-009-9866-7

     • Google Scholar
187. 1. Wray KB

     (2010) Selection and predictive success

     Erkenntnis 72:365–377.

     https://doi.org/10.1007/s10670-009-9206-6

     • Google Scholar
188. 1. Yarkoni T

     (2020) The generalizability crisis

     Behavioral and Brain Sciences 45:1–37.

     https://doi.org/10.1017/S0140525X20001685

     • PubMed
     • Google Scholar
189. 1. Yoshihara M
     2. Yoshihara M

     (2018) “Necessary and sufficient” in biology is not necessarily necessary - confusions and erroneous conclusions resulting from misapplied logic in the field of biology, especially neuroscience

     Journal of Neurogenetics 32:53–64.

     https://doi.org/10.1080/01677063.2018.1468443

     • PubMed
     • Google Scholar
190. 1. Zamora-López G
     2. Zhou C
     3. Kurths J

     (2010) Cortical hubs form a module for multisensory integration on top of the hierarchy of cortical networks

     Frontiers in Neuroinformatics 4:1.

     https://doi.org/10.3389/neuro.11.001.2010

     • PubMed
     • Google Scholar

Author details

1. Mika Rubinov

   1. Departments of Biomedical Engineering, Computer Science, and Psychology, Vanderbilt University, Nashville, United States
   2. Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, United States

   Contribution

   Conceptualization, Data curation, Formal analysis, Writing - original draft, Writing - review and editing

   For correspondence

   mika.rubinov@vanderbilt.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-4787-7075

• 5,107

  views

• 634

  downloads

• 5

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.