In the last decades, the idea that people routinely and implicitly predict upcoming words during language comprehension turned from a highly controversial hypothesis to a widely accepted assumption. Initial objections to prediction in language were based on a lack of empirical support (e.g. Zwitserlood, 1989), incompatibility with traditional bottom-up models and contemporary interactive models of language comprehension (e.g. Kintsch, 1988; Marslen-Wilson and Tyler, 1980), and the purported futility of prediction in a generative system where sentences can continue in infinitely many different ways (Jackendoff, 2002). Current theories of language comprehension, however, reject such objections and posit prediction as an integral and inevitable mechanism by which comprehension proceeds quickly and incrementally (e.g. Altmann and Mirković, 2009; Dell and Chang, 2014; Pickering and Garrod, 2013). Prediction, that is, context-based pre-activation of an upcoming linguistic input, is thought to occur at all levels of linguistic representation (semantic, morpho-syntactic and phonological/orthographic) and serves to facilitate the integration of newly available bottom-up information into the unfolding sentence- or discourse-representation. In this line of thought, language is yet another domain in which the brain acts as a prediction machine (Clark, 2013; Van Berkum, 2010; see also Friston, 2005, 2010; Summerfield and de Lange, 2014), hard-wired to continuously match sensory inputs with top-down, grammatical or probabilistic expectations based on context and memory.

What promoted linguistic prediction from outlandish and deeply contentious to ubiquitous and somewhat anodyne? One of the key and most acclaimed pieces of empirical evidence for linguistic prediction to date comes from a landmark Nature Neuroscience publication by DeLong et al. (2005), whose approach exploited a phonological rule of English whereby the indefinite article is realized as a before consonant-initial words and as an before vowel-initial words. In their experiment, participants read sentences of varying degree of contextual constraint that led to expectations for a particular consonant- or vowel-initial noun. This expectation was operationalized as a word’s cloze probability (cloze), calculated in a separate, non-speeded sentence completion task as the percentage of continuations of a sentence fragment with that word (Taylor, 1953). For example, the sentence fragment "The day was breezy so the boy went outside to fly...” is continued with ‘a’ by 86% of participants, and "The day was breezy so the boy went outside to fly a..." is continued with ‘kite’ by 89% of participants. In the main experiment, word-by-word sentence presentation enabled DeLong and colleagues to examine electrical brain activity elicited by articles that were concordant with the highly expected but yet unseen noun (‘a’, followed by ‘kite’), or by articles that were incompatible with the highly expected noun and heralded a less expected one (‘an’, followed by ‘airplane’). Of note, an unexpected like ‘a/an’ does not rule out that the expected noun appears, just that it appears as the immediately following word (e.g., ‘an old kite’), we return to this issue in the Discussion. The dependent measure was the amplitude of the N400 Event-related potential (ERP), a negative ERP deflection that peaks approximately 400 ms after word onset and is maximal at centroparietal electrodes (Kutas and Hillyard, 1980). The N400 is elicited by every word of an unfolding sentence, and its amplitude is smaller (less negative) with increasing ease of semantic processing (Kutas and Hillyard, 1984). In this article, we use ‘N400 amplitude’ as a shorthand for ‘ERP amplitude in the time window associated with the N400’; this ERP amplitude is actually a sum of the N400 ERP component and other ERP components (reflecting other aspects of cognition) that overlap with it in time and space. DeLong et al. found that the N400 amplitude for a given word decreased as a function of increasing cloze probability, both for nouns and, critically, for articles. DeLong et al. presented the systematic and graded N400 modulation by article-cloze as strong evidence that participants activated the nouns and articles in advance of their appearance, and that the disconfirmation of this prediction by the less-expected articles resulted in processing difficulty (higher N400 amplitude at the article).

The results obtained with this elegant design warranted a much stronger conclusion than related results available at the time. Previous studies that employed a visual-world paradigm had revealed listeners’ anticipatory eye-movements toward visual objects on the basis of probabilistic or grammatical considerations (e.g. Altmann and Kamide, 1999). However, predictions in such studies are scaffolded onto already-available visual context, and therefore do not measure purely pre-activation, but perhaps re-activation of word information previously activated by the visual object itself (Huettig, 2015). DeLong and colleagues examined brain responses to information associated with concepts that were not pre-specified and had to be retrieved from long-term memory ‘on-the-fly’. Furthermore, DeLong and colleagues were the first to muster evidence for highly specific pre-activation of a word’s phonological form, rather than merely its semantic (e.g. Federmeier and Kutas, 1999) or morpho-syntactic features (e.g. Van Berkum et al., 2005; Wicha et al., 2004). Crucially, as their demonstration involved semantically identical articles (function words) rather than nouns or adjectives (content words) that are rich in meaning, the observed N400 modulation by article-cloze is unlikely to reflect difficulty interpreting the articles themselves. Most notably, DeLong and colleagues were the first to examine brain activity elicited by a range of more- or less-predictable articles, not simply most- versus least-expected. Based on the observed correlation, they argued that pre-activation is not all-or-none and limited to highly constraining contexts, but occurs in a graded, probabilistic fashion, with the strength of a word pre-activation proportional to its cloze probability. Moreover, they concluded that prediction is an integral part of real-time language processing and, most likely, a mechanism for propelling the comprehension system to keep up with the rapid pace of natural language.

DeLong et al.’s study has had an immense impact on the field of psycholinguistics, neurolinguistics and beyond. It is cited by authoritative reviews (e.g. Altmann and Mirković, 2009; Hagoort, 2017; Lau et al., 2008; Pickering and Clark, 2014; Pickering and Garrod, 2007) as delivering decisive evidence for probabilistic prediction of words all the way up to their phonological form. Moreover, as a demonstration of pre-activation of phonological form (sound) during reading, it is sometimes cited as evidence for ‘prediction through production’ (e.g. Pickering and Garrod, 2013), the hypothesis that linguistic predictions are implicitly generated by the language production system. To date, DeLong et al. has received a total of 766 citations (Google Scholar), averaging to more than one citation per week over the past decade, with an increasing number of citations in each subsequent year. The results also played an important role in settling an ongoing debate in the neuroscience of language. It provided the clearest evidence that the N400 component, which some researchers had long taken to directly index the high-level compositional processes by which people integrate a word’s meaning with its context (Brown and Hagoort, 1993; Chwilla et al., 1995; Connolly and Phillips, 1994; Friederici et al., 1999; van Berkum et al., 1999; Van Petten et al., 1999), actually reflected non-compositional processes by which word information is accessed as a function of context (e.g. Kutas and Hillyard, 1984).

But how robust are gradient effects of form prediction? In over a decade that has passed since the publication by DeLong and colleagues, there is still no published study that directly replicates their graded pattern of results (for an overview, see Ito et al., 2017b). DeLong and colleagues also performed an alternative analysis of the same data, using cloze as a categorical variable instead of a continuous variable. This analysis did not yield a statistically significant result (p.59 in DeLong, 2009) and was not mentioned in the published report. In at least three other unpublished data sets (DeLong, 2009; Miyamoto, 2016), DeLong and colleagues did not find a significant correlation between article-N400 and cloze probability. Martin et al. (2013) claimed a successful conceptual replication in native speakers of English but not in bilinguals. However, their study did not test for a graded effect of cloze, and differed from the original in many crucial aspects of the experimental design, data-preprocessing and statistical analysis, clouding both a qualitative and quantitative comparison to the original results. Moreover, two attempts to replicate the Martin et al. results in English monolinguals failed to yield a reliable effect of cloze on article-ERPs (Ito et al., 2017b); for results that combined data from monolinguals and bilinguals, see Ito et al., 2017a).

As the tremendous scientific impact of the DeLong et al. findings is at odds with the apparent lack of replication attempts, we report here a direct replication study. Inspired by recent demonstrations for the need for large subject-samples in psychology and neuroscience research (Button et al., 2013; Open Science Collaboration, 2015), our replication spanned nine laboratories each with a sample size equal to or greater than that of the original. In addition to duplicating the original analysis, our replication attempt also seeks to improve upon DeLong et al.’s data analysis. DeLong et al.’s original analysis reduced an initial pool of 2560 data points (32 subjects who each read 80 sentences) to 10 grand-average values, by averaging N400 responses over trials within 10 cloze probability decile-bins (cloze 0–10, 11–20, etc.), per participant and then averaging over participants, even though these bins held greatly different numbers of observations (for example, the 0–10 cloze bin contained 37.5% of all data, whereas the 90–100 cloze bin contained only about 4%, which means that the reliability of the estimates per bin greatly differ, increasing the likelihood of obtaining spurious results; for additional discussion see Ito et al., 2017b). These 10 values were correlated with the average cloze value per bin, yielding numerically high correlation coefficients with large confidence intervals (e.g., the Cz electrode showed a statistically significant r-value of 0.68 with a 95% confidence interval ranging from 0.09 to 0.92). However, this analysis potentially compromises power by discretizing cloze probability into deciles and not distinguishing various sources of subject-, item-, bin-, and trial-level variation. Furthermore, treating subjects as a fixed rather than random factor potentially inflates false positive rates, since the overall cloze effect is confounded with by-subject variation in the effect (Barr et al., 2013; Clark, 1973).

In our replication study, we followed two pre-registered analysis routes: a replication analysis that duplicated the DeLong et al. analysis, and a single-trial analysis that modelled variance at the level of item and subject (with a linear mixed-effects model), which offers better control over false-positives than the replication analysis when analyzing effects of the continuous predictor cloze probability. The effect of cloze on noun-elicited N400s (DeLong et al., 2015; Kutas and Hillyard, 1984) is necessary but not sufficient evidence for the claim on pre-activation in language processing (as it is also compatible with the view that the noun’s cloze probability correlates with the ease of integration of that noun into the context). It serves as a manipulation check to ensure that the experiment is able to successfully detect graded variation in N400 amplitude, but does not provide strong evidence for the prediction of phonological form. That evidence would come from the ERPs elicited by articles. Observing a reliable effect of cloze on article-elicited N400s in the replication analysis and, in particular, in the single-trial analysis, would constitute powerful evidence for the pre-activation of phonological form during reading.

We first obtained offline cloze probabilities for all target articles and nouns from a group of native English speakers. These values closely resembled those of the original study (see Methods for details). In the subsequent ERP experiment, a different group of participants (N = 334) read the sentences word-by-word from a computer display at a rate of 2 words per second, while we recorded their electrical brain activity at the scalp. The replication analysis and single-trial analysis described below were each pre-registered at https://osf.io/eyzaq/.

Replication analysis

We sorted the articles and nouns into 10 bins based on each word’s cloze probability (e.g. items with 0–10% cloze were put in one bin, 10–20% in another, etc.). For each laboratory, we averaged ERPs per bin first within, then across, participants. No baseline correction was used, following the procedure described in the Methods section in DeLong et al. (2005). We then correlated the averaged cloze values per bin with mean ERP amplitude in the N400 time window (200–500 ms) elicited by the nouns (for the noun analysis) or articles (for the article analysis) from the corresponding bin, yielding a Pearson correlation coefficient (r-value) per EEG channel. This analysis yielded a very different pattern than DeLong et al. observed (Figure 1). In no laboratory did article-N400 amplitude at centro-parietal sites become significantly smaller (less negative) as article-cloze probability increased (in fact, in most laboratories the pattern went into the opposite direction). Only in one laboratory (Lab 2) did the correlation coefficient have a p-value below .05 in the predicted direction (positive) at any electrode (uncorrected for multiple comparisons), but this effect was observed at a few left-frontal electrodes, not at the central-parietal electrodes where DeLong et al. found their N400 effects. Moreover, in two laboratories (Labs 3 and 5), a statistically significant effect was observed in the opposite direction, larger (more negative) article-N400 amplitude for articles with increasing cloze probability. For the nouns, the pattern was more similar to the DeLong et al. results. In six laboratories (Lab 2, 3, 4, 6, 7, and 9), noun-N400 amplitude for nouns at central-parietal or parietal-occipital electrodes became smaller with increasing noun-cloze, and in two other laboratories (Lab 5 and 8) the effects clearly went in the expected direction without reaching statistical significance.

Figure 1

Download asset Open asset

Figure 1—source data 1

r-values for the articles for each laboratory and each channel

https://doi.org/10.7554/eLife.33468.003

Download elife-33468-fig1-data1-v1.txt

Figure 1—source data 2

p-values for the articles for each laboratory and each channel.

https://doi.org/10.7554/eLife.33468.004

Download elife-33468-fig1-data2-v1.txt

Figure 1—source data 3

r-values for the nouns for each laboratory and each channel.

https://doi.org/10.7554/eLife.33468.005

Download elife-33468-fig1-data3-v1.txt

Figure 1—source data 4

r-values for the nouns for each laboratory and each channel.

https://doi.org/10.7554/eLife.33468.006

Download elife-33468-fig1-data4-v1.txt

DeLong et al. recently mentioned using a 500 ms baseline correction procedure that was not mentioned in the published study (personal communication by DeLong, March 2017). In an exploratory analysis, we therefore recomputed the correlations based on data pooled from all laboratories using this baseline correction procedure (Figure 2). This analysis also showed a lack of statistically significant positive correlations for the articles, but statistically significant positive correlations for the nouns. In exploratory Bayesian analyses reported below, we perform an analysis to establish whether these results are consistent with the size and direction of the effects reported by DeLong et al., regardless of statistical significance.

Figure 2

Download asset Open asset

Figure 2—source data 1

r-values for the articles for each channel, computed across laboratories. 

https://doi.org/10.7554/eLife.33468.008

Download elife-33468-fig2-data1-v1.txt

Figure 2—source data 2

p-values for the articles for each channel, computed across laboratories.

https://doi.org/10.7554/eLife.33468.009

Download elife-33468-fig2-data2-v1.txt

Figure 2—source data 3

r-values for the nouns for each channel, computed across laboratories. 

https://doi.org/10.7554/eLife.33468.010

Download elife-33468-fig2-data3-v1.txt

Figure 2—source data 4

p-values for the nouns for each channel, computed across laboratories. 

https://doi.org/10.7554/eLife.33468.011

Download elife-33468-fig2-data4-v1.txt

Single-trial analysis

We first performed baseline correction by subtracting the average amplitude in the 100 ms time window before word onset. Baseline-corrected ERPs for relatively expected and unexpected words and difference waveforms are shown in Figure 3. Then, for the data pooled across all laboratories, we used linear mixed effects models to regress the N400 amplitude (in a spatiotemporal region of interest selected a priori based on the DeLong et al. results) on cloze probability. For the articles, the effect of cloze was not statistically significant at the α = 0.05 level, β = 0.29, CI [−0.08, .67], χ²(1)=2.31, p=0.13 (see Figure 4, left panel) , with β referring to the N400 difference in microvolts associated with stepping from 0% to 100% cloze. Unless otherwise indicated, p-values are two-tailed, and CIs are two-tailed 95% confidence intervals. The effect of cloze on N400 amplitude at the article did not significantly differ between laboratories, χ²(8)=7.90, p=0.44. For the nouns, however, higher cloze values were strongly associated with smaller N400s, β = 2.22, CI [1.76, 2.69], χ²(1)=56.50, p<0.001 (see Figure 4, right panel). This pattern did not significantly differ between laboratories, χ²(8)=11.59, p=0.17. The effect of cloze on noun-N400s was statistically different from its effect on article-N400s, χ²(1)=31.38, p<0.001.

Figure 3

Download asset Open asset

Figure 4

Download asset Open asset

Exploratory (i.e. not pre-registered) single-trial analyses

The effect of article-cloze did not significantly vary as a function of subject comprehension question accuracy, χ²(1)=0.45, p=0.50. In addition, the effect of article-cloze was also not statistically significant when subject comprehension accuracy was included in the analysis (100 ms baseline: β = 0.24, CI [−0.17, .64], χ²(1)=1.27, p=0.26).

In our dataset, an analysis in the 500 to 100 ms time window before article-onset revealed a non-significant effect of cloze that resembled the pattern observed after article-onset, β = 0.16, CI [−0.07, .39], χ²(1)=1.82, p=0.18 (Figure 5). Because the sentence context of each item was identical for the expected and unexpected article, effects in the pre-article window cannot be meaningfully related to the appearance of the article. Effects in this window must therefore be due to a spurious mix of ‘residual EEG background noise’ (activity that differed between expected and unexpected conditions but was unrelated to actual expectancy) with EEG activity associated with the specific word appearing before the article (which varied between items in terms of lexical characteristics, contextual constraint, and sentence position). The observed result in this time window therefore suggests that a 500 ms baseline correction procedure, which was used but not reported in DeLong et al. (2005), would better correct for pre-article voltage-levels. We repeated our analysis with the 500 ms baseline correction procedure. Compared to the article-cloze effect observed in the pre-registered analysis, the observed effect with the new baseline procedure (Figure 5) was numerically smaller and yielded a higher p-value (β = 0.14, CI [−0.25, .53], χ²(1)=0.46, p=0.50).

Figure 5

Download asset Open asset

Upon request of reviewers for this journal, we also performed an additional exploratory analysis with cloze as a dichotomous variable (based on a medium-split, thus disregarding the known variability in cloze values). We note that this type of analysis was not reported in DeLong et al. (2005), although it was reported in the corresponding thesis chapter (DeLong, 2009) and did not yield a statistically significant effect of cloze on article-elicited ERPs. We performed this analysis for articles (100 and 500 ms baseline correction) and nouns. The results did not change substantially, and, in fact, each analysis yielded a lower χ² value (and higher p-value) for the cloze variable than the corresponding analysis with cloze as a continuous predictor. The results can be reproduced from our online dataset and code.

Exploratory Bayesian analyses

For the articles, our pre-registered replication analyses yielded non-significant p-values, indicating failure to reject the null-hypothesis that cloze has no effect on N400 activity. To better adjudicate between the null-hypothesis (H₀) and an alternative hypothesis (H_r), we performed an exploratory replication Bayes factor analysis for correlations (Wagenmakers et al., 2016). The obtained replication Bayes factor quantifies the evidence that there is an effect in the size and direction reported by DeLong et al. (see Figure 6). For the articles, this yielded strong to extremely strong evidence for the null hypothesis that the effect of cloze is zero, with BF_0r values up to 154 (at the Cz electrode depicted by DeLong et al., BF_0r = 77), and strongest evidence at the posterior channels. For the nouns, we obtained extremely strong evidence for the alternative hypothesis that the effect is consistent with the original effect, particularly at posterior channels, with BF₁₀ values up to 9,163,515 (at Cz, BF_r0 = 10,725). The pattern of results was similar when the 500 ms pre-stimulus baseline correction was applied.

Figure 6

Download asset Open asset

Next, we computed Bayesian mixed-effect model estimates (β) and 95% credible intervals (CrI) for our single-trial analyses, using priors based on the results from DeLong et al. In both of our article-analyses, credible intervals included zero (100 ms baseline: β = 0.31, CrI [−0.06 .69]; 500 ms baseline: β = 0.17, CrI [−0.22 .55]). For the nouns, zero was not within the credible interval: β = 2.24, CrI [1.77 2.70]. The analyses suggest that the data (combined with prior assumptions about the effect) are not very consistent with the hypothesis that the article-effect is zero (further information and posterior summaries are available in Figure 7), but also are extremely inconsistent with the hypothesis that the article-effect is as big as that observed by DeLong et al. (2005). The data are most consistent with an effect that is more likely to be positive than zero or negative but is very small (so small that it was not detected at traditional significance levels in this large-scale experiment with substantially higher power than previous experiments).

Figure 7

Download asset Open asset

Control experiment

Lack of a statistically significant, article-elicited prediction effect could reflect a general insensitivity of our participants to the phonologically conditioned variation of the English indefinite article, that is, a/an alternation. We ruled out this alternative explanation in an additional experiment that followed the replication experiment as part of the same experimental session. Participants read 80 short sentences containing the same nouns as the replication experiment, preceded by a phonologically licit or illicit article (e.g. ‘David found a/an apple...”), presented in the same manner as before. In each laboratory, nouns following illicit articles elicited a late positive-going waveform compared to nouns following licit articles (see Figure 8), starting at about 500 ms after word onset and strongest at parietal electrodes. This standard P600 effect (Osterhout and Holcomb, 1992) was confirmed in a single-trial analysis, χ²(1)=83.09, p<0.001, and did not significantly differ between labs, χ²(8)=8.98, p=0.35.

Figure 8

Download asset Open asset

1. 1. Nieuwland M

   (2018) Replication Recipe Analysis plan

   Available at the Open Science Framework.

   https://osf.io/eyzaq/

1. 1. Altmann GT
   2. Kamide Y

   (1999) Incremental interpretation at verbs: restricting the domain of subsequent reference

   Cognition 73:247–264.

   https://doi.org/10.1016/S0010-0277(99)00059-1

   • PubMed
   • Google Scholar
2. 1. Altmann GT
   2. Mirković J

   (2009) Incrementality and prediction in human sentence processing

   Cognitive Science 33:583–609.

   https://doi.org/10.1111/j.1551-6709.2009.01022.x

   • PubMed
   • Google Scholar
3. 1. Baayen RH
   2. Davidson DJ
   3. Bates DM

   (2008) Mixed-effects modeling with crossed random effects for subjects and items

   Journal of Memory and Language 59:390–412.

   https://doi.org/10.1016/j.jml.2007.12.005

   • Google Scholar
4. 1. Baggio G
   2. Hagoort P

   (2011) The balance between memory and unification in semantics: A dynamic account of the N400

   Language and Cognitive Processes 26:1338–1367.

   https://doi.org/10.1080/01690965.2010.542671

   • Google Scholar
5. 1. Barr DJ
   2. Levy R
   3. Scheepers C
   4. Tily HJ

   (2013) Random effects structure for confirmatory hypothesis testing: Keep it maximal

   Journal of Memory and Language 68:255–278.

   https://doi.org/10.1016/j.jml.2012.11.001

   • PubMed
   • Google Scholar
6. 1. Brothers T
   2. Swaab TY
   3. Traxler MJ

   (2017) Goals and strategies influence lexical prediction during sentence comprehension

   Journal of Memory and Language 93:203–216.

   https://doi.org/10.1016/j.jml.2016.10.002

   • Google Scholar
7. 1. Brown C
   2. Hagoort P

   (1993) The processing nature of the n400: evidence from masked priming

   Journal of Cognitive Neuroscience 5:34–44.

   https://doi.org/10.1162/jocn.1993.5.1.34

   • PubMed
   • Google Scholar
8. 1. Brown CM
   2. Hagoort P
   3. Chwilla DJ

   (2000) An event-related brain potential analysis of visual word priming effects

   Brain and Language 72:158–190.

   https://doi.org/10.1006/brln.1999.2284

   • PubMed
   • Google Scholar
9. 1. Button KS
   2. Ioannidis JP
   3. Mokrysz C
   4. Nosek BA
   5. Flint J
   6. Robinson ES
   7. Munafò MR

   (2013) Power failure: why small sample size undermines the reliability of neuroscience

   Nature Reviews Neuroscience 14:365–376.

   https://doi.org/10.1038/nrn3475

   • PubMed
   • Google Scholar
10. 1. Chwilla DJ
    2. Brown CM
    3. Hagoort P

    (1995) The N400 as a function of the level of processing

    Psychophysiology 32:274–285.

    https://doi.org/10.1111/j.1469-8986.1995.tb02956.x

    • PubMed
    • Google Scholar
11. 1. Clark A

    (2013) Whatever next? Predictive brains, situated agents, and the future of cognitive science

    The Behavioral and Brain Sciences 36:181–204.

    https://doi.org/10.1017/S0140525X12000477

    • PubMed
    • Google Scholar
12. 1. Clark HH

    (1973) The language-as-fixed-effect fallacy: A critique of language statistics in psychological research

    Journal of Verbal Learning and Verbal Behavior 12:335–359.

    https://doi.org/10.1016/S0022-5371(73)80014-3

    • Google Scholar
13. 1. Connolly JF
    2. Phillips NA

    (1994) Event-related potential components reflect phonological and semantic processing of the terminal word of spoken sentences

    Journal of Cognitive Neuroscience 6:256–266.

    https://doi.org/10.1162/jocn.1994.6.3.256

    • PubMed
    • Google Scholar
14. 1. Dell GS
    2. Chang F

    (2014) The P-chain: relating sentence production and its disorders to comprehension and acquisition

    Philosophical Transactions of the Royal Society B: Biological Sciences 369:20120394.

    https://doi.org/10.1098/rstb.2012.0394

    • PubMed
    • Google Scholar
15. 1. DeLong KA
    2. Urbach TP
    3. Kutas M

    (2005) Probabilistic word pre-activation during language comprehension inferred from electrical brain activity

    Nature Neuroscience 8:1117–1121.

    https://doi.org/10.1038/nn1504

    • PubMed
    • Google Scholar
16. Book

    1. DeLong KA
    2. Urbach TP
    3. Kutas M

    (2017)

    Concerns with Nieuwland et al. Multi-Lab Study (2017)

    San Diego: University of California.

    • Google Scholar
17. Book

    1. DeLong KA

    (2009)

    Electrophysiological Explorations of Linguistic Pre-Activation and Its Consequences During Online Sentence Processing

    San Diego: University of California.

    • Google Scholar
18. 1. Federmeier KD
    2. Kutas M

    (1999) A rose by any other name: long-term memory structure and sentence processing

    Journal of Memory and Language 41:469–495.

    https://doi.org/10.1006/jmla.1999.2660

    • Google Scholar
19. 1. Friederici AD
    2. Steinhauer K
    3. Frisch S

    (1999) Lexical integration: sequential effects of syntactic and semantic information

    Memory & Cognition 27:438–453.

    https://doi.org/10.3758/BF03211539

    • PubMed
    • Google Scholar
20. 1. Friston K

    (2005) A theory of cortical responses

    Philosophical Transactions of the Royal Society B: Biological Sciences 360:815–836.

    https://doi.org/10.1098/rstb.2005.1622

    • PubMed
    • Google Scholar
21. 1. Friston K

    (2010) The free-energy principle: a unified brain theory?

    Nature Reviews Neuroscience 11:127–138.

    https://doi.org/10.1038/nrn2787

    • PubMed
    • Google Scholar
22. Website

    1. Gelman A
    2. Loken E

    (2013) The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time

    http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf
23. 1. Gelman A
    2. Stern H

    (2006) The difference between “significant” and “not significant” is not itself statistically significant

    The American Statistician 60:328–331.

    https://doi.org/10.1198/000313006X152649

    • Google Scholar
24. 1. Hagoort P

    (2017) The core and beyond in the language-ready brain

    Neuroscience & Biobehavioral Reviews 81:194–204.

    https://doi.org/10.1016/j.neubiorev.2017.01.048

    • PubMed
    • Google Scholar
25. 1. Hauk O
    2. Davis MH
    3. Ford M
    4. Pulvermüller F
    5. Marslen-Wilson WD

    (2006) The time course of visual word recognition as revealed by linear regression analysis of ERP data

    NeuroImage 30:1383–1400.

    https://doi.org/10.1016/j.neuroimage.2005.11.048

    • PubMed
    • Google Scholar
26. 1. Huettig F

    (2015) Four central questions about prediction in language processing

    Brain Research 1626:118–135.

    https://doi.org/10.1016/j.brainres.2015.02.014

    • PubMed
    • Google Scholar
27. 1. Ito A
    2. Corley M
    3. Pickering MJ
    4. Martin AE
    5. Nieuwland MS

    (2016) Predicting form and meaning: Evidence from brain potentials

    Journal of Memory and Language 86:157–171.

    https://doi.org/10.1016/j.jml.2015.10.007

    • Google Scholar
28. 1. Ito A
    2. Martin AE
    3. Nieuwland MS

    (2017a) How robust are prediction effects in language comprehension? Failure to replicate article-elicited N400 effects

    Language, Cognition and Neuroscience 32:954–965.

    https://doi.org/10.1080/23273798.2016.1242761

    • Google Scholar
29. 1. Ito A
    2. Martin AE
    3. Nieuwland MS

    (2017b) Why the A/AN prediction effect may be hard to replicate: a rebuttal to Delong, Urbach, and Kutas (2017)

    Language, Cognition and Neuroscience 32:974–983.

    https://doi.org/10.1080/23273798.2017.1323112

    • Google Scholar
30. Book

    1. Jackendoff R

    (2002) Foundations of Language: Brain, Meaning, Grammar, Evolution

    Oxford; New York: Oxford University Press.

    https://doi.org/10.1093/acprof:oso/9780198270126.001.0001

    • Google Scholar
31. 1. Jung TP
    2. Makeig S
    3. Humphries C
    4. Lee TW
    5. McKeown MJ
    6. Iragui V
    7. Sejnowski TJ

    (2000) Removing electroencephalographic artifacts by blind source separation

    Psychophysiology 37:163–178.

    https://doi.org/10.1111/1469-8986.3720163

    • PubMed
    • Google Scholar
32. 1. Kintsch W

    (1988) The role of knowledge in discourse comprehension: a construction-integration model

    Psychological Review 95:163–182.

    https://doi.org/10.1037/0033-295X.95.2.163

    • PubMed
    • Google Scholar
33. 1. Kutas M
    2. Federmeier KD

    (2011) Thirty years and counting: finding meaning in the N400 component of the event-related brain potential (ERP)

    Annual Review of Psychology 62:621–647.

    https://doi.org/10.1146/annurev.psych.093008.131123

    • PubMed
    • Google Scholar
34. 1. Kutas M
    2. Hillyard SA

    (1980) Reading senseless sentences: brain potentials reflect semantic incongruity

    Science 207:203–205.

    https://doi.org/10.1126/science.7350657

    • PubMed
    • Google Scholar
35. 1. Kutas M
    2. Hillyard SA

    (1984) Brain potentials during reading reflect word expectancy and semantic association

    Nature 307:161–163.

    https://doi.org/10.1038/307161a0

    • PubMed
    • Google Scholar
36. 1. Lau EF
    2. Holcomb PJ
    3. Kuperberg GR

    (2013) Dissociating N400 effects of prediction from association in single-word contexts

    Journal of Cognitive Neuroscience 25:484–502.

    https://doi.org/10.1162/jocn_a_00328

    • PubMed
    • Google Scholar
37. 1. Lau EF
    2. Namyst A
    3. Fogel A
    4. Delgado T

    (2016) A direct comparison of n400 effects of predictability and incongruity in adjective-noun combination

    Collabra 2:.

    https://doi.org/10.1525/collabra.40

    • Google Scholar
38. 1. Lau EF
    2. Phillips C
    3. Poeppel D

    (2008) A cortical network for semantics: (de)constructing the N400

    Nature Reviews Neuroscience 9:920–933.

    https://doi.org/10.1038/nrn2532

    • PubMed
    • Google Scholar
39. 1. Luck SJ
    2. Gaspelin N

    (2017) How to get statistically significant effects in any ERP experiment (and why you shouldn't)

    Psychophysiology 54:146–157.

    https://doi.org/10.1111/psyp.12639

    • PubMed
    • Google Scholar
40. 1. Marslen-Wilson W
    2. Tyler LK

    (1980) The temporal structure of spoken language understanding

    Cognition 8:1–71.

    https://doi.org/10.1016/0010-0277(80)90015-3

    • PubMed
    • Google Scholar
41. 1. Martin CD
    2. Thierry G
    3. Kuipers J-R
    4. Boutonnet B
    5. Foucart A
    6. Costa A

    (2013) Bilinguals reading in their second language do not predict upcoming words as native readers do

    Journal of Memory and Language 69:574–588.

    https://doi.org/10.1016/j.jml.2013.08.001

    • Google Scholar
42. Website

    1. Miyamoto K

    (2016) Hemispheric differences in linguistic prediction given high constraint contexts

    Presentation Given at the Kutas Cognitive Electrophysiology Lab. Accessed April 23, 2016.

    https://www.slideshare.net/KianaMiyamoto/kutas-lab-latart
43. 1. Nieuwland MS
    2. Martin AE

    (2012) If the real world were irrelevant, so to speak: The role of propositional truth-value in counterfactual sentence comprehension

    Cognition 122:102–109.

    https://doi.org/10.1016/j.cognition.2011.09.001

    • PubMed
    • Google Scholar
44. Preprint

    1. Nieuwland MS
    2. Politzer-Ahles S
    3. Heyselaar E
    4. Segaert K
    5. Darley E
    6. Kazanina N
    7. Von Grebmer Zu Wolfsthurn S
    8. Bartolozzi F
    9. Kogan V
    10. Ito A
    11. Mézière D
    12. Barr DJ
    13. Rousselet G
    14. Ferguson HJ
    15. Busch-Moreno S
    16. Fu X
    17. Tuomainen J
    18. Kulakova E
    19. Husband EM
    20. Donaldson DI
    21. Kohút Z
    22. Rueschemeyer SA
    23. Huettig F

    (2017) Limits on prediction in language comprehension: A multi-lab failure to replicate evidence for probabilistic pre-activation of phonology

    BioRxiv.

    https://www.biorxiv.org/content/early/2017/02/25/111807

    • Google Scholar
45. 1. Nieuwland MS

    (2016) Quantification, prediction, and the online impact of sentence truth-value: Evidence from event-related potentials

    Journal of Experimental Psychology: Learning, Memory, and Cognition 42:316–334.

    https://doi.org/10.1037/xlm0000173

    • PubMed
    • Google Scholar
46. 1. O'Regan K

    (1979) Saccade size control in reading: evidence for the linguistic control hypothesis

    Perception & Psychophysics 25:501–509.

    https://doi.org/10.3758/BF03213829

    • PubMed
    • Google Scholar
47. 1. Open Science Collaboration

    (2015) PSYCHOLOGY. Estimating the reproducibility of psychological science

    Science 349:aac4716.

    https://doi.org/10.1126/science.aac4716

    • PubMed
    • Google Scholar
48. 1. Osterhout L
    2. Holcomb PJ

    (1992) Event-related brain potentials elicited by syntactic anomaly

    Journal of Memory and Language 31:785–806.

    https://doi.org/10.1016/0749-596X(92)90039-Z

    • Google Scholar
49. 1. Otten M
    2. Nieuwland MS
    3. Van Berkum JJ

    (2007) Great expectations: specific lexical anticipation influences the processing of spoken language

    BMC Neuroscience 8:89.

    https://doi.org/10.1186/1471-2202-8-89

    • PubMed
    • Google Scholar
50. 1. Otten M
    2. Van Berkum JJ

    (2009) Does working memory capacity affect the ability to predict upcoming words in discourse?

    Brain Research 1291:92–101.

    https://doi.org/10.1016/j.brainres.2009.07.042

    • PubMed
    • Google Scholar
51. 1. Otten M
    2. Van Berkum JJA

    (2008) Discourse-based word anticipation during language processing: prediction or priming?

    Discourse Processes 45:464–496.

    https://doi.org/10.1080/01638530802356463

    • Google Scholar
52. 1. Pickering MJ
    2. Clark A

    (2014) Getting ahead: forward models and their place in cognitive architecture

    Trends in Cognitive Sciences 18:451–456.

    https://doi.org/10.1016/j.tics.2014.05.006

    • PubMed
    • Google Scholar
53. 1. Pickering MJ
    2. Garrod S

    (2007) Do people use language production to make predictions during comprehension?

    Trends in Cognitive Sciences 11:105–110.

    https://doi.org/10.1016/j.tics.2006.12.002

    • PubMed
    • Google Scholar
54. 1. Pickering MJ
    2. Garrod S

    (2013) An integrated theory of language production and comprehension

    Behavioral and Brain Sciences 36:329–347.

    https://doi.org/10.1017/S0140525X12001495

    • PubMed
    • Google Scholar
55. 1. Smith NJ
    2. Levy R

    (2013) The effect of word predictability on reading time is logarithmic

    Cognition 128:302–319.

    https://doi.org/10.1016/j.cognition.2013.02.013

    • PubMed
    • Google Scholar
56. 1. Steinhauer K
    2. Royle P
    3. Drury JE
    4. Fromont LA

    (2017) The priming of priming: Evidence that the N400 reflects context-dependent post-retrieval word integration in working memory

    Neuroscience Letters 651:192–197.

    https://doi.org/10.1016/j.neulet.2017.05.007

    • PubMed
    • Google Scholar
57. 1. Summerfield C
    2. de Lange FP

    (2014) Expectation in perceptual decision making: neural and computational mechanisms

    Nature Reviews Neuroscience 15:745–756.

    https://doi.org/10.1038/nrn3838

    • PubMed
    • Google Scholar
58. 1. Tanner D
    2. Morgan-Short K
    3. Luck SJ

    (2015) How inappropriate high-pass filters can produce artifactual effects and incorrect conclusions in ERP studies of language and cognition

    Psychophysiology 52:997–1009.

    https://doi.org/10.1111/psyp.12437

    • PubMed
    • Google Scholar
59. 1. Taylor WL

    (1953) “Cloze Procedure”: A New Tool for Measuring Readability

    Journalism Bulletin 30:415–433.

    https://doi.org/10.1177/107769905303000401

    • Google Scholar
60. 1. Van Berkum JJ
    2. Brown CM
    3. Zwitserlood P
    4. Kooijman V
    5. Hagoort P

    (2005) Anticipating upcoming words in discourse: evidence from ERPs and reading times

    Journal of Experimental Psychology: Learning, Memory, and Cognition 31:443–467.

    https://doi.org/10.1037/0278-7393.31.3.443

    • PubMed
    • Google Scholar
61. 1. van Berkum JJ
    2. Hagoort P
    3. Brown CM

    (1999) Semantic integration in sentences and discourse: evidence from the N400

    Journal of Cognitive Neuroscience 11:657–671.

    https://doi.org/10.1162/089892999563724

    • PubMed
    • Google Scholar
62. 1. Van Berkum JJ

    (2010)

    The brain is a prediction machine that cares about good and bad-any implications for neuropragmatics?

    Italian Journal of Linguistics 22:181–208.

    • Google Scholar
63. 1. Van Petten C
    2. Coulson S
    3. Rubin S
    4. Plante E
    5. Parks M

    (1999) Time course of word identification and semantic integration in spoken language

    Journal of Experimental Psychology: Learning, Memory, and Cognition 25:394–417.

    https://doi.org/10.1037/0278-7393.25.2.394

    • PubMed
    • Google Scholar
64. 1. Van Petten C
    2. Luka BJ

    (2012) Prediction during language comprehension: benefits, costs, and ERP components

    International Journal of Psychophysiology 83:176–190.

    https://doi.org/10.1016/j.ijpsycho.2011.09.015

    • PubMed
    • Google Scholar
65. Preprint

    1. Vasishth S
    2. Mertzen D
    3. Jäger LA
    4. Gelman A

    (2018) The statistical significance filter leads to overoptimistic expectations of replicability

    Psyarxiv.

    https://arxiv.org/abs/1702.00556

    • Google Scholar
66. 1. Wagenmakers EJ
    2. Verhagen J
    3. Ly A

    (2016) How to quantify the evidence for the absence of a correlation

    Behavior Research Methods 48:413–426.

    https://doi.org/10.3758/s13428-015-0593-0

    • PubMed
    • Google Scholar
67. 1. Wicha NY
    2. Moreno EM
    3. Kutas M

    (2004) Anticipating words and their gender: an event-related brain potential study of semantic integration, gender expectancy, and gender agreement in Spanish sentence reading

    Journal of Cognitive Neuroscience 16:1272–1288.

    https://doi.org/10.1162/0898929041920487

    • PubMed
    • Google Scholar
68. 1. Wlotko EW
    2. Federmeier KD

    (2012) So that's what you meant! Event-related potentials reveal multiple aspects of context use during construction of message-level meaning

    NeuroImage 62:356–366.

    https://doi.org/10.1016/j.neuroimage.2012.04.054

    • PubMed
    • Google Scholar
69. 1. Wlotko EW
    2. Federmeier KD

    (2015) Time for prediction? The effect of presentation rate on predictive sentence comprehension during word-by-word reading

    Cortex 68:20–32.

    https://doi.org/10.1016/j.cortex.2015.03.014

    • PubMed
    • Google Scholar
70. Preprint

    1. Yan S
    2. Kuperberg GR
    3. Jaeger TF

    (2017) Prediction (Or Not) during language processing. a commentary on nieuwland, et al.(2017) And Delong, et al.(2005)

    bioRxiv.

    https://doi.org/10.1101/143750

    • Google Scholar
71. 1. Zwitserlood P

    (1989) The locus of the effects of sentential-semantic context in spoken-word processing

    Cognition 32:25–64.

    https://doi.org/10.1016/0010-0277(89)90013-9

    • PubMed
    • Google Scholar

Thank you for submitting your article "Large-scale replication study reveals a limit on probabilistic prediction in language comprehension" for consideration by eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Sabine Kastner as the Senior Editor. The following individual involved in review of your submission have agreed to reveal his identity: Matt Davis (Reviewer #3).

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission. The three reviewers were unanimous in recognizing the importance of this large-scale replication (with pre-registered hypotheses, across multiple labs) study, and of publishing this kind of work. They all agreed that the study is appropriate for eLife, but also all agreed that the presentation requires some polishing before publication.

The positives here are clear. While no experimental evidence can "prove" the null hypothesis, this work highlights that the field has adopted a very strong assumption about mechanisms of language comprehension even though the landmark evidence for this idea is not very robust. The reviewers all applaud the effort that went into this large-scale, multi-site study. All hope that the publication of this study encourages others to undertake this kind of effort, which not only is principled, but provides a large dataset that can fuel additional statistical analysis, computational modelling, and theoretical debate. The reviewers felt that emphasizing these issues even more in the manuscript would make clear how this kind of work contributes to the literature.

Each of the three reviewers provided extraordinarily long and detailed comments that were thoughtful and constructive. While we have tried to consolidate these comments, the result remains a quite long discussion, for which I apologize; however, given the care that went into these reviews, I decided it was helpful to preserve much of the content.

Essential revisions:

1) At the highest level, the theoretical importance of the original paper-and the current replication- needs to be even more clearly laid out. The interpretation of the original results is stated in only vague terms, which leads to confusion about what one should conclude. Crucially, the authors never really spell what mechanism would lead to effects on the N400 in response to an article preceding the putatively predicted noun (as opposed to the N1, P2, LAN, P600, or any other ERP component).

The current paper seems to argue for the "high-level compositional" (Introduction) view of the N400. The "N400 is elicited by every word of an unfolding sentence, and its amplitude is smaller (less negative) with increasing ease of semantic processing" (Introduction). Therefore, if a word does not fit semantically well within a context, N400 effects are expected. But DeLong et al., (2005) argued that the N400 reflects lexico-semantic prediction. The mixing of these two views is problematic here, as the DeLong et al., (2005) design "involved semantically identical articles (function words) rather than nouns or adjectives (content words) that are rich in meaning" (Introduction), which in turn would make "the observed N400 modulation by article-cloze.… unlikely to reflect difficulty interpreting the articles themselves" (Introduction). Unfortunately, this feature of the study makes it hard to make a precise prediction about what to expect at the level of processing the definite articles, especially if one is espousing a semantic integration view of the N400, for several reasons:

a) N400 effects are normally investigated in content words, not closed class words like articles.

b) As closed class words, articles elicit different ERP morphologies than open class words, including within the N400 time window.

c) Previous studies contrasting ERP responses to articles that are compatible or incompatible in upcoming word gender features elicited differences in late positivities, not N400.

d) The control study in the current manuscript shows that the incorrect pairing of a definite article allomorph with a subsequent word induces later positivities, not N400 effects. This calls into question why we should expect an N400 effect in the case of the predicted upcoming word.

e) There is a long history of research tying the N400 to general lexico-semantic processing, but not compositional semantics. At different places, the authors seem to argue both of these views. Under the compositional semantic integration view, the N400 may be affected by the phonological form of articles. Both forms contribute similarly to the semantics of the sentence (they are allomorphs after all). Semantically reversible sentences do not show N400 effects even when one order of the arguments is semantically odd (e.g.," the hunters chased the fox" vs. "the fox chased the hunters"), which argues against a compositional view of the N400. Given all this, why should one expect the N400 to be affected for allomorphic variations of the definite article when no obvious semantic integration effects are involved?

f) The fact that the a/an alternation is not triggered by the predicted head noun, but rather by the word immediately following the article, also confuses things quite a bit: why should there be, under any interpretation of the N400, an N400 effect for the alternating articles on the basis of the predicted head noun if the head noun is not necessarily (or even statistically) the trigger for the article form alternation? While the authors discuss this idea, it casts serious doubt on the original interpretation of the DeLong et al., findings.

These issues call into question the theoretical importance of the original study (and thus the replication). Contrary to what the authors state in the paper, that original paper offers only weak evidence for a "strong prediction view" precisely because the theoretical implications are vague and unspecified.

2) Throughout the paper, the authors emphasize the failure to find graded neural effects of cloze probability on the indefinite article preceding a noun. The reviewers do not believe this is as critical as it is played up in the manuscript. For instance, the current study also provides some evidence that a mismatch between articles (a/an) and subsequent nouns, conducted in the same participants tested in a subsequent experiment, leads to different neural activity. Moreover, a gradient in response at the level of group data (as in the original study) could nonetheless arise from discrete, all-or-none prediction at the single-trial level. It seems a bit disingenuous to argue that the study should call into question the importance of prediction on theories of language comprehension so broadly.

Relatedly, it would be good if results from studies of other "predictive" effects were considered and reconciled with the present findings. The authors list several of these in the Discussion section, but omit others; the authors should provide a balanced discussion of other evidence of different prediction effects, such as

- gender predictions (Wicha, Moreno and Kutas, 2004; Otten, Nieuwland and van Berkum, 2008; Van Berkum et al., 2005).

- predictions of initial sounds of upcoming words (Connolly and Phillips, 1994).

- expectations of the form of words with specific syntactic classes (Dikker et al., 2010).

- processes affected pre-N400 ERP responses driven by form-level characteristics (Lau et al., 2006).

- effects on early sensory responses like the M100 (Dikker et al., 2010; Dikker, Rabagliati, and Pylkkanen, 2009).

- behavioral evidence by close-shadowing (Marslen-Wilson, 1985).

The authors argue that the present null results for articles can "be straightforwardly reconciled with effects reported for other pre-nominal manipulations" (Discussion section), but studies like those above provide evidence for predictive processing. The preceding paragraph (Discussion section) offer a number of caveats, but readers of this paragraph might take away the message that the current results "do not necessarily exclude phonological form pre-activation" rather than the message from previous page (Discussion section) that "there is currently no clear evidence to support routine probabilistic pre-activation of a noun's phonological form during sentence comprehension.

In general, the wording of the manuscript should be revised throughout so as to not overstate the implications of the findings, while at the same time giving a consistent message about what the findings do mean.

3) The correlation analysis in the original paper analyzed only 10 data points, each of which is a mean across 32 participants who each did ~80 trials (binned into deciles based on cloze probability). This analysis excludes several critical sources of variance between trials, items, and participants, which increases the likelihood of type I error. On the other hand, the analysis is also underpowered, since with only 10 data points and 9 degrees of freedom, statistical significance requires an exceptionally strong correlation; thus, the likelihood of type II error is great (making it not very surprising that results fail to replicate). This is an important lesson for the field and should be brought our more clearly in the present manuscript. While adding additional analysis is not necessary for the manuscript to be publishable, this point could be strengthened, for instance, by running simulations to assess the power of the original DUK study, varying the number of participants and items and exploring whether a graded effect of cloze probability on article and noun processing can be detected.

4) On a related note, the DUK study also reports a reliable difference in N400 magnitude between high and low cloze probability articles and nouns (Figure 1A of DUK, left panel); their analysis included by-participant variation as a random effect. Given their counterbalanced design and arguments from Raaijmakers et al., (1999), this analysis is less prone to errors than their correlational analysis. The categorical analysis provides additional evidence for a role of prediction – albeit not a graded one. Figure 3 of the present manuscript presents similar categorical results, but there is no statistical analysis reported. Do these new results also show significant evidence for a non-graded prediction of articles? If so, this would substantially modify the conclusions drawn from the current study. This should be considered and discussed.

5. The reviewers noted that there is an online critique and re-analysis posted on BioRxiv by Yan, Kuperberg and Jaeger (https://www.biorxiv.org/content/early/2017/05/30/143750) that endorses the importance of the issues raised and the value of the data collected here. It's not clear whether and how the authors anticipated this commentary in the present manuscript.

For instance, this commentary suggests that Bayesian surprisal, KL divergence between pre- and post-article word probability distributions, or other variables might show a closure correspondence to the neural data than does cloze probability; both the current authors and the original study make an implicit assumption that cloze probability is the best predictor.

Another example is the idea raised in the commentary that there is a chain of cognitive processes mediating between meaning predictions and form-based predictions for indefinite articles. A failure, probabilistic limitation, or probability mis-estimation (e.g., cloze tests overestimating the probability of an indefinite article) in these mediating processes may explain why the strongly constraining lexical/semantic predictions set-up by the sentence contexts don't always influence the processing of indefinite articles. The sentence contexts set up semantic expectations, which constrain the choice of lemmas in critical sentence positions. Some predicted lemmas have phonological forms that have specific consequences for articles; however, form-based predictions may not follow even if a semantically predicted lemma is chosen (e.g., "a kite" vs "an aeroplane", but also "an old kite" or "a plane"). The authors acknowledge this point (Discussion section), but do not carry this line of argument through the rest of the paper. It's not the case that a failure to observe predictions at the level of the phonological form of indefinite articles necessarily implies that all prediction is absent; yet in a number of places they seem to argue along these lines.

The manuscript's impact would be heightened by incorporating comments that address points like these that were raised in this online critique.

6) It is great that the authors included Bayes factor analyses to gain theoretical insights from null results. These analyses provide evidence in favor of the null hypothesis that there are no graded predictions for articles, despite numerical effects in the expected direction. However, these analyses depend on assumptions about the expected effect size and other details that are not clearly spelled out. These priors should be provided. For instance, Figure 1A/B DUK suggests that the expected effect size of the cloze probability effect on the article is considerably smaller than the effect on the noun itself. It seems that this was taken into account in Bayes factor computations (given the graphs in SI Figure 2), but what other assumptions are made in this analysis? Was the variance expected to be equivalent for articles and nouns? Can the authors justify the choice of using a Gaussian? What does the estimated effect size being greater than zero mean for the "pre article" period (in SI Figure 2)? If predictive processing occurs, then predictions should have been computed prior to the onset of the article.

7) The use of a pre-article baseline correction may be problematic, since neural responses prior to the onset of the article will include processing of linguistic information that contributes to the generation of predictions or (equivalently) constrains the likely form and meaning of upcoming words. For example, recent data from Grisoni et al., (2017) suggests a "semantic readiness potential" reflecting pre-activation of specific semantic representations. Similar semantic pre-activation is likely present in the sentences used in the current study. By doing a pre-article baseline, such differences could corrupt or conflate with differences ascribed to neural responses to the article. The authors consider but do not describe or discuss these effects in Supplementary figure 2. While the authors are not responsible for the fact that the original study used an unconventional baseline, in subsection “Single-trial analysis” the manuscript implies that there is a reason why it is appropriate, because the current authors observed non-significant cloze effects immediately prior to the critical noun.

At a minimum, this potential confound should be raised and discussed. Better still, the authors could run an analysis in which they either use a pre-sentence baseline or analyze raw ERPs without baseline correction and compare to the pre-article baseline analysis. They could also analyze whether there are differences to highly vs weakly constraining sentences prior to the onset of the article (building on the analysis in SI Figure 2), an approach that might benefit from using a different predictor variable the captures the strength of prediction (e.g. the entropy of the cloze probability distribution) rather than not whether or not the prediction is subsequently confirmed or violated. Inclusion of these types of additional analyses could support a more nuanced discussion of whether or not participants compute the likely meaning of upcoming words.

8) The distinction between form and meaning predictions may influence differences or similarities between N400 effects seen on articles and on nouns. The original DUK paper reports similar magnitude and topography of N400 effects on both the article and noun. It would be instructive to compare the location, timing, and effect size for cloze probability correlations between the article and the noun in the current, larger dataset. Additional tests for difference between the significant and null correlations would be valuable. For instance, is the reliable effect of graded cloze probability on nouns significantly different from the null effect of cloze probability on articles? This difference should be reliable if semantic integration is the key driver of N400 effects on nouns.

In a similar vein, what is the assumed model of the N400 generator? There are at least two published computational models that seek to explain the large body of N400 effects in the existing ERP literature. One proposes something akin to integration demands (Cheyette and Plaut, 2016), another proposes a form of (semantic) prediction error (Rabovsky and McRae, 2014). Such models provide a helpful context in which to discuss whether or not prediction violations at the level of form will lead to an N400 in the absence of any ongoing semantic integration demands. Ultimately, questions about the replicability or otherwise of the data presented by DUK are only informative to the extent that they are or, are not theoretically constraining. A discussion of these and other computational theories of the N400 are important for interpreting the present results, and how these results should be considered in the field of language comprehension going forward.

Other general comments for the authors to consider:

1) The authors conducted a single-trial analysis using linear mixed effects regression (LMER) analysis. This is the most appropriate analysis method since it includes both between-participant and between-item variance and combines all of the usable data from individual participants and single trials into one analysis. While LMER analysis is now preferred to correlation analysis used in the original study, it only came to prominence after the publication of the original paper. Pointing out the advantages of LMER analyses would enhance the current paper's impact. For instance, especially when analyzing effects of continuous predictor variables, LMER may offer better control over false-positives than conventional analyses. This is an important message for researchers studying the neuroscience of language, who generally do not undertake this approach.

2) The flip side of using single-trial LMER, however, is that it is not clear whether or not the link between cloze probability and neural responses is linear or logistic. The independent measure (cloze probability) used as a predictor variable is summarizing a binary outcome variable (i.e., did individual participants in the cloze test generate the article "a" or "an") as a probability. It is at least plausible that neural outcomes could be similarly binary: participants predict "a" or "an" and then don (or don't) generate a neural prediction error when an unexpected word is presented. Would reliable effects of cloze probability be observed with a logistic or binomial linking function? While this is not the form of graded prediction tested by DUK, it seems quite likely that averaging over many participants and many trials (as in the DUK correlation analysis) would generate a graded, linear relationship between cloze probability and neural outcomes, even if the underlying relationship was logistic or non-linear. This possibility should be considered, assessed and discussed.

3) Previous work has explored the presence/absence of fillers, which is another difference between the DUK study and the current one. While it is unlikely to explain the failure to replicate results, some additional discussion seems appropriate. Given the large scale of the present dataset there should be many opportunities for determining whether cloze probability effects (for nouns, if not for articles) are enhanced once participants have already encountered a number of distinctive, high/low cloze probability sentences.

4) The fact that the nine labs inadvertently manipulated the timing of stimulus presentations of the critical articles and nouns (see footnote 3, Materials and methods section) is unlikely critical to their findings/conclusions. However, it would be useful to know whether the timing of early, visual form responses for written words that are delayed and non-delayed differ. Simply arguing that the change in timing is "unlikely to be noticed" and "after the N400 window" is making assumptions that can and should be tested with their ERP data.

5) The use of different baselines was confusing to multiple reviewers. A table that summarizes details of all the current analyses and how these correspond to the procedures used in the original DUK study would help the reader keep straight the different results.

6) Both the original and current studies present a high percentage of a/an continuations in the cloze test. Could this procedure bias participants towards using a/an rather than "his", "hers", "their," etc. that aren't marked for the phonological form of the subsequent noun? After encountering several sentences in which the indefinite article is appropriate, participants may routinely complete the remaining sentences with an indefinite article continuation due to this sort of syntactic priming. A comment about this would be welcome.

7) In multiple places, the authors use phrasing about how the study argues for "a more limited role for prediction during language comprehension." This wording seems overly broad, conflating very different conceptions of prediction across different levels of representation. There is a deep theoretical division between proposed mechanisms of conceptual pre-activation (e.g., the concept of 'readiness' in the discourse literature, linked to the N400 literature beautifully in Jos Van Berkum's work) and mechanisms of predicting the form of the upcoming input (commitment to a particular linguistic formulation of the message). It seems likely that the former, long argued for in the discourse literature, might play a central role in language comprehension, while contributions from the prediction-of-form mechanism may be quite limited. It would be unfortunate if readers take the current study to cast doubt on both extremely different mechanisms, when it actually only speaks to the latter mechanism. Please consider rewording this throughout.

Essential revisions:

1) […] These issues call into question the theoretical importance of the original study (and thus the replication). Contrary to what the authors state in the paper, that original paper offers only weak evidence for a "strong prediction view" precisely because the theoretical implications are vague and unspecified.

We thank the reviewer for these comments. In the Introduction we now state “DeLong et al., presented the systematic, graded N400 modulation by article-cloze as strong evidence that participants activated the nouns and articles in advance of their appearance, and that the disconfirmation of this prediction by the less-expected articles resulted in processing difficulty (higher N400 amplitude at the article)”

This reviewer questions the theoretical relevance offered by DeLong et al., about their own findings and points out different interpretations of the N400 that may have different repercussions for interpreting the DUK results. However, that interpretation exercise assumes that the original effects were reliable and replicable. The motivation of our study was to test whether the effects can be replicated in the first place, after which a theoretical interpretation becomes relevant, as we offer in our discussion. This reviewer suggests that we are advocating a particular interpretation of the N400. This is not the case. Our group does not take the N400 as a pure measure of high-level semantic composition. We purposefully used the neutral term “ease of semantic processing”, which can include both memory access and integration processes. Our only assumption regarding the N400 followed the logic of DeLong et al., namely that noun-elicited N400s themselves do not yield unambiguous evidence for prediction.

We certainly agree that DUK offers only weak evidence for the “strong prediction view”, in particular because their effects are weak, and they tried multiple analyses. We now refer to the original results throughout the paper as “the most acclaimed” instead of “the strongest evidence”. This is because while DUK may offer only weak evidence, that is not how their results have been and are still received in the wider literature: indeed, several impactful review papers literally cite DUK as ‘strong evidence’. This reviewer may call into question the theoretical relevance of the original study, but, as we describe in our text, the original study has had a major impact on the field, is cited in all the major theoretical review papers on linguistic prediction. Moreover, never has a question been raised about the replicability or interpretation of the DUK findings (with a few exceptions which we cite in the text).

This reviewer also states that all pre-nominal gender manipulations elicited effects other than N400 effects. This is not the case. Prenominal gender manipulations have elicited a range of effects, including N400 effects (e.g. Otten and Berkum, 2009; Wicha et al., 2003; Kochari and Flecken, under review) and sometimes P600-like effects or frontal negative ERP effects. We mentioned this in the previous submission and have now expanded this section. However, why different effects are observed for those gender-manipulations is unknown and beyond the scope of our paper on a different manipulation.

2) Throughout the paper, the authors emphasize the failure to find graded neural effects of cloze probability on the indefinite article preceding a noun. The reviewers do not believe this is as critical as it is played up in the manuscript. For instance, the current study also provides some evidence that a mismatch between articles (a/an) and subsequent nouns, conducted in the same participants tested in a subsequent experiment, leads to different neural activity. Moreover, a gradient in response at the level of group data (as in the original study) could nonetheless arise from discrete, all-or-none prediction at the single-trial level. It seems a bit disingenuous to argue that the study should call into question the importance of prediction on theories of language comprehension so broadly.

We added comments to clarify that we do not question the general importance of prediction in theories of language (see also our response to the next parts of comment 2). Here and in M4, the reviewers bring up the use of cloze as a dichotomous variable. We have performed this analysis with a median-split approach (100 and 500 ms baseline for the articles, 100 ms baseline for the nouns). The new code is also available on our OSF page. Importantly, the results are not affected substantially. Moreover, these analyses all yielded lower chi-sq values (and higher p-values) than the continuous approach. We mention these analyses and their results in the Results section.

Relatedly, it would be good if results from studies of other "predictive" effects were considered and reconciled with the present findings. The authors list several of these in the Discussion section, but omit others; the authors should provide a balanced discussion of other evidence of different prediction effects, such as

- gender predictions (Wicha, Moreno and Kutas, 2004; Otten, Nieuwland and van Berkum, 2008; Van Berkum, et al., 2005).

- predictions of initial sounds of upcoming words (Connolly and Phillips, 1994).

- expectations of the form of words with specific syntactic classes (Dikker, et al., 2010).

- processes affected pre-N400 ERP responses driven by form-level characteristics (Lau et al., 2006).

- effects on early sensory responses like the M100 (Dikker et al., 2010; Dikker, Rabagliati, and Pylkkanen, 2009).

- behavioral evidence by close-shadowing (Marslen-Wilson, 1985).

We thank the reviewer for the suggestion. We already cite many of the relevant papers on gender manipulations, which are the most relevant because they involve prenominal manipulations like DUK. We mentioned that these studies have not elicited a consistent pattern of results and have sometimes demonstrated non-replicability. However, providing a review of all ERP effects associated with prediction is beyond the scope of our work, and there are already many other papers who review this literature. This reviewer appears to take all the listed studies as clear-cut evidence for some form of prediction. But many of these studies are problematic for all sorts of reasons, including the issue of replicability and data-contingent analysis. For example, the Connolly and Phillips results are intriguing but rely entirely on the distinction between the N200 and N400, which is poorly defined in the original study, and not observed in many other studies (Van Petten et al., 1999; Van den Brink et al.,; Boudewyn et al., 2015; Diaz and Swaab, 2007). The study by Lau et al., reported ELAN-results that were not necessarily pre-N400 (200-400 ms), could only be obtained with a common average procedure that no other ELAN study has ever used, showed clear evidence of a ‘polar average reference effect’, and has failed to replicate (Kaan, Kirkham and Wijnen, 2016). The Dikker et al., studies are low N (~12) studies and p-values in the.01-.05 range with many researcher degrees of freedom due to the methodology, and do not show any M1 modulation by predictability. We also note that all the studies mentioned by the reviewer are single studies that, like DUK, have not been directly replicated, hence the robustness of the reported findings needs to be verified, as well as their generalizability to other contexts, labs, etc. In other words, all these observations are worthy of a dedicated, thorough and critical review, but that is for a different paper. Our aim is not to question the general role of prediction in language comprehension (see next point), but to illustrate the importance of replication.

The authors argue that the present null results for articles can "be straightforwardly reconciled with effects reported for other pre-nominal manipulations" (Discussion section), but studies like those above provide evidence for predictive processing. The preceding paragraph (Discussion section) offer a number of caveats, but readers of this paragraph might take away the message that the current results "do not necessarily exclude phonological form pre-activation" rather than the message from previous page (Discussion section) that "there is currently no clear evidence to support routine probabilistic pre-activation of a noun's phonological form during sentence comprehension

In general, the wording of the manuscript should be revised throughout so as to not overstate the implications of the findings, while at the same time giving a consistent message about what the findings do mean.

We do not question the existence of predictive processing in general, and we now make that more explicit in the revised Discussion section(“Our results, however, should not be taken as evidence against prediction in language processing more generally, and we believe that prediction could play an important role in language comprehension”). We also note that each of these studies argues for a slightly different aspect of prediction, and none of them clearly supports "routine probabilistic pre-activation of a noun's phonological form during written sentence comprehension". Furthermore, as pointed out above, these studies are not as strong as they may first appear, for all sorts of reasons. As for our own results, they are not necessarily evidence against phonological form pre-activation, just not very clear evidence for it; that is what we would like readers to take away from our discussion.

3) The correlation analysis in the original paper analyzed only 10 data points, each of which is a mean across 32 participants who each did ~80 trials (binned into deciles based on cloze probability). This analysis excludes several critical sources of variance between trials, items, and participants, which increases the likelihood of type I error. On the other hand, the analysis is also underpowered, since with only 10 data points and 9 degrees of freedom, statistical significance requires an exceptionally strong correlation; thus, the likelihood of type II error is great (making it not very surprising that results fail to replicate). This is an important lesson for the field and should be brought our more clearly in the present manuscript. While adding additional analysis is not necessary for the manuscript to be publishable, this point could be strengthened, for instance, by running simulations to assess the power of the original DUK study, varying the number of participants and items and exploring whether a graded effect of cloze probability on article and noun processing can be detected.

We appreciate the suggestion to emphasize the potential problems with the correlation approach. The problems with this approach are clear. In our study, for example, while a statistically significant effect at the noun was found in each participating lab using a more powerful single-trial analysis, the effect across labs in the correlation analysis was mixed. We have now added a statement to that effect in the Discussion section (“Moreover, our single-trial analysis revealed a significant noun-cloze effect in each of the laboratories, further demonstrating that our single-trial analysis is a more powerful approach than the averaged-based correlation approach of DeLong et al.”). Given the obvious problems with the correlation-analysis, we feel that focusing too much on that analysis would detract from our own single-trial results. In addition, we feel that running simulations to assess the power of DUK with varying participant and item numbers is unnecessarily burdensome, if all that is available to us is the correlation coefficients: running such simulations using the actual DUK data would be a lot more meaningful, but they have refused to share their data so far. In addition, we know that DUK have analyzed their own data with a single-trial analysis, which is a more informative analysis given the weaknesses of the original analysis, and which also did not yield a statistically significant article-effect. We assume that DeLong et al., will publish those new results in the near future, so that power-analyses can be applied to the improved analyses.

4) On a related note, the DUK study also reports a reliable difference in N400 magnitude between high and low cloze probability articles and nouns (Figure 1A of DUK, left panel); their analysis included by-participant variation as a random effect. Given their counterbalanced design and arguments from Raaijmakers et al., (1999), this analysis is less prone to errors than their correlational analysis. The categorical analysis provides additional evidence for a role of prediction – albeit not a graded one. Figure 3 of the present manuscript presents similar categorical results, but there is no statistical analysis reported. Do these new results also show significant evidence for a non-graded prediction of articles? If so, this would substantially modify the conclusions drawn from the current study. This should be considered and discussed.

DUK figure 1a showed the difference between high and low only as “illustrative ERPs”, the original report did NOT report any categorical statistical analysis. However, DeLong et al., (2009) DID report such an analysis and the effect of cloze on article-elicited ERPs was not significant. We have also performed these analyses, as described in M2.

5) The reviewers noted that there is an online critique and re-analysis posted on BioRxiv by Yan, Kuperberg and Jaeger (https://www.biorxiv.org/content/early/2017/05/30/143750) that endorses the importance of the issues raised and the value of the data collected here. It's not clear whether and how the authors anticipated this commentary in the present manuscript.

For instance, this commentary suggests that Bayesian surprisal, KL divergence between pre- and post-article word probability distributions, or other variables might show a closure correspondence to the neural data than does cloze probability; both the current authors and the original study make an implicit assumption that cloze probability is the best predictor.

We are aware of this unpublished critique and re-analysis by YKJ, which is a comment on a previous draft of our manuscript that became available on Biorxiv in February 2017. We did not incorporate discussion of this work because we felt that a separate, dedicated response was more appropriate, in which we could extensively comment on some of the details of their analysis and their argumentation (after all, their commentary is 60 pages long). YKJ bring up an interesting point, namely that the cloze variable should be log-transformed, based on their account in terms of Bayesian surprisal, which itself is not a new argument (e.g. Smith and Levy, 2013). We have now added a brief discussion of the Yan et al. commentary. To retain a balance between succinctness and completeness in the main text, we moved our fuller response to an online supplement on our OSF page with a link in the text (https://osf.io/mb2ud/).

In essence, the YKJ conclusion is not different from what we conclude, namely that there is some evidence for a non-zero effect. However, YKJ ignore our strong evidence that the effect is not like that of DUK, and misrepresent our conclusions to argue their case (e.g., “their conclusion that their data provides no evidence for prediction on the article”, while we in fact purposefully refrained from using terms like “no evidence”, we used terms like “no convincing evidence” or “no clear evidence”). The YKJ argument about how cloze probabilities should be analyzed is ultimately one for further testing, because their exploratory analysis was conditional on our data. YKJ applaud us for “leading by example in pre-registering their study” but, have yet to follow our example. It is one year since we published our pre-print, and about 10 months after Yan et al., performed their analyses. As far as we are aware, they have not proceeded to test their hypothesis in a pre-registered study, nor have they attempted to reanalyze the wealth of N400 data from their own labs. YKJ report no independent data to support the conclusion that Bayesian surprisal is a better predictor of N400 activity than cloze probability. Unlike YKJ, we have run their transformed analysis on other cloze datasets. The results speak against a generally better fit of log-cloze, and we have also made those data available online.

The YKJ analysis did yield a significant article-cloze effect in our data, and a model with log-transformed cloze had a lower AIC (an indication of model fit) than a model with regular cloze. For the nouns, the transformation yielded a slightly higher T value for cloze. According to one of the current reviewers (S11): “No valid conclusion can be drawn from a change in numerical magnitude or a change in the observed p-value”, and if that is true the same concern should apply to YKJ as well. The article-effect becomes significant whereas it was not-significant before, but the difference between significant and not-significant itself may not be significant (Gelman and Stern, 2006). Moreover, their analysis could be problematic because between-item differences in the lower cloze scale are not matched on relevant variables. This is also suggested by the fact that their analysis also amplifies the effect in the pre-article time window (a marginally significant effect at p=0.058), so it might just be ‘amplifying noise’ (by amplifying the between-item differences at the lower end of the cloze scale), and does not yield a significant effect when we used the original DUK baseline. Their results also hinge entirely on the zero-cloze values which they argue cannot be reliably estimated. Jaeger mentioned in a private message that “all the signal is in the zero-cloze values”, and this is correct; in fact, if one repeats their analysis without zero-cloze values the effect size changes direction (higher cloze leads to more negative voltage). In other words, the transformation seems to matter very little, and what it does seem to do is boost the impact of zero-cloze values which are problematic because they are indeterminate, so the evidence for their case is not very convincing.

YKJ present the significant effect of article-cloze in their main text, and they put the analyses that do not yield as clear support in a footnote, and do not report analysis of the pre-article window. This seems somewhat disingenuous to us, given that we reported all analyses in the main text, significant or non-significant. YKJ also report their analysis as “the only one we performed”, although there is no record of this and cannot be verified. In fact, private twitter messages by Jaeger to Nieuwland stated the opposite: that multiple analyses (to deal with the zero-cloze smoothing issue) were tried, and although they were said to lead to similar results, those are not mentioned in YKJ, and the reported ‘correction for multiple comparisons’ can therefore not be complete. A similar concern can be raised against the other analyses that were tried: for instance, YKJ analyzed the data with and without lab as random effect and they analyzed the data to test effects in each individual lab (with and without log transformation, so 2*9=18 comparisons). In none of those analyses did they find a statistically significant effect, despite what they state (“we find a significant effect on both the noun and the article in 8 out of 9 of the labs that participated in the replication attempt”). In addition, none of these analyses were taken into account during the correction for multiple comparisons. All this merely demonstrates the need for YKJ to specify and pre-register their analysis in advance and run a confirmatory study to test their hypothesis.

Finally, what strikes us as odd is that YKJ have many things to say about why our analysis is sub-optimal and our conclusions should not be accepted. But when it comes to the DUK data, all they say is “For the purpose of this discussion, we take the effect that they report at face value”, despite the many problems with the DUK analysis that we had identified. The importance of the log-transform as suggested by YKJ would also need to apply to the Delong study, and it's not clear that the transform there would strengthen their effect rather than weaken it.

Another example is the idea raised in the commentary that there is a chain of cognitive processes mediating between meaning predictions and form-based predictions for indefinite articles. A failure, probabilistic limitation, or probability mis-estimation (e.g., cloze tests overestimating the probability of an indefinite article) in these mediating processes may explain why the strongly constraining lexical/semantic predictions set-up by the sentence contexts don't always influence the processing of indefinite articles. The sentence contexts set up semantic expectations, which constrain the choice of lemmas in critical sentence positions. Some predicted lemmas have phonological forms that have specific consequences for articles; however, form-based predictions may not follow even if a semantically predicted lemma is chosen (e.g., "a kite" vs "an aeroplane", but also "an old kite" or "a plane"). The authors acknowledge this point (Discussion section), but do not carry this line of argument through the rest of the paper. It's not the case that a failure to observe predictions at the level of the phonological form of indefinite articles necessarily implies that all prediction is absent; yet in a number of places they seem to argue along these lines.

The manuscript's impact would be heightened by incorporating comments that address points like these that were raised in this online critique.

We have added an explicit statement that our results are not to be taken as evidence against prediction in general (see response to Essential revision #2); we never said that our results suggest ‘prediction is absent’ and it was not our intention to imply this. As this reviewer writes, our submission already mentioned that the article-form does not rule out the expected meaning altogether. This aspect of the a/an rule in part triggered our interest in the DUK study, which we take up in the Discussion section. We have added a mention of how the a/an rule works on page 4 but we do not see why this would have to be brought up “throughout the rest of the paper”. We wrote it as a potential explanation for the discrepant findings, and this was available in our initial pre-print (available online in Feb 2017), and was also discussed earlier in Ito, Martin and Nieuwland (2017; available online in May 2016), so all in advance of the Yan et al. commentary.

6) It is great that the authors included Bayes factor analyses to gain theoretical insights from null results. These analyses provide evidence in favor of the null hypothesis that there are no graded predictions for articles, despite numerical effects in the expected direction. However, these analyses depend on assumptions about the expected effect size and other details that are not clearly spelled out. These priors should be provided. For instance, Figure 1A/B DUK suggests that the expected effect size of the cloze probability effect on the article is considerably smaller than the effect on the noun itself. It seems that this was taken into account in Bayes Factor computations (given the graphs in SI Figure 2), but what other assumptions are made in this analysis? Was the variance expected to be equivalent for articles and nouns? Can the authors justify the choice of using a Gaussian?

This reviewer may have missed our description of the priors in the methods section. Yes, they differed for articles and nouns because they were based on the DeLong correlation results. We have seen the LMEM results of the DUK data, but DeLong et al. refused to share their LMEM based priors for our analysis. We can inform the reviewer that those priors pulled our estimates closer to zero. Yes, variance was assumed to be equivalent. There is no ground to suspect a Gaussian would not be appropriate. Of note, these are all exploratory, non-preregistered analyses and the code and data are online so that anyone can explore our data further.

What does the estimated effect size being greater than zero mean for the "pre article" period (in SI Figure 2)? If predictive processing occurs, then predictions should have been computed prior to the onset of the article.

In short, we don’t really know. We analyzed this window because we found out, after publishing our pre-print, that the 500 ms window was used for baseline correction by DeLong. Their choice differed from most of the other studies in their lab, so maybe they picked this window to correct for pre-article differences, we do not know. We also saw what looked like slow drift effects in the ERPs in some labs in this window. The effects in this window cannot be meaningfully interpreted. This is because the two versions of each item are identical before the article. So, any effect there must be a mix of noise and effects due to differences between the sentence contexts (e.g., the predictability or integrability of the pre-article word, sentence position, lexical effects; all of these were not matched between the sentence contexts). It is possible that the distributions of the unexpected and expected articles along the cloze scale is somehow interacting with some of these between-item sentence context differences. We agree that predictions should be generated before the article, but this particular design does not allow for a meaningful analysis.

7) The use of a pre-article baseline correction may be problematic, since neural responses prior to the onset of the article will include processing of linguistic information that contributes to the generation of predictions or (equivalently) constrains the likely form and meaning of upcoming words. For example, recent data from Grisoni et al., (2017) suggests a "semantic readiness potential" reflecting pre-activation of specific semantic representations. Similar semantic pre-activation is likely present in the sentences used in the current study. By doing a pre-article baseline, such differences could corrupt or conflate with differences ascribed to neural responses to the article. The authors consider but do not describe or discuss these effects in Supplementary figure 2. While the authors are not responsible for the fact that the original study used an unconventional baseline, in subsection “Single-trial analysis” the manuscript implies that there is a reason why it is appropriate, because the current authors observed non-significant cloze effects immediately prior to the critical noun.

At a minimum, this potential confound should be raised and discussed. Better still, the authors could run an analysis in which they either use a pre-sentence baseline or analyze raw ERPs without baseline correction and compare to the pre-article baseline analysis. They could also analyze whether there are differences to highly vs weakly constraining sentences prior to the onset of the article (building on the analysis in SI Figure 2), an approach that might benefit from using a different predictor variable the captures the strength of prediction (e.g. the entropy of the cloze probability distribution) rather than not whether or not the prediction is subsequently confirmed or violated. Inclusion of these types of additional analyses could support a more nuanced discussion of whether or not participants compute the likely meaning of upcoming words.

We appreciate this concern about using a baseline window period in which predictive processing takes place. The reviewer correctly points out an inherent problem with choosing an appropriate baseline for the article in a study that searches for the presence of prediction for the article. We performed the requested analysis without baseline correction, this does not change the observed pattern (below) and introduced a large amount of variance into the data that a baseline procedure is supposed to remove.

Fixed effects:

Estimate Std. Error df t value Pr(>|t|)cloze-effect 0.1351 0.3897 474.5000 0.347 0.729

We emphasize that we are not arguing against some form of semantic prediction, and our revised draft makes this more clear. While this reviewer wants additional exploratory analyses to support a discussion of whether or not word meaning was anticipated, that was never the question of our study, and there is already quite a lot of evidence that people predict the meaning of upcoming words. Furthermore, the 80 sentence contexts are not matched on relevant variables known to impact EEG activity, so the experimental design is not suitable for addressing the question of pre-article constraint. We considered it important to make this clearer to the readers. In subsection “Exploratory (i.e., not pre-registered) single-trial analyses” we added “Because the sentence context of each item was identical for the expected and unexpected article, effects in the pre-article window cannot be meaningfully related to the appearance of the article. Effects in this window must therefore be due to a spurious mix of ‘residual EEG background noise’ (activity that differed between expected and unexpected conditions but was unrelated to actual expectancy) with EEG activity associated with the specific word appearing before the article (which varied between items in terms of lexical characteristics, contextual constraint, and sentence position).”

8) The distinction between form and meaning predictions may influence differences or similarities between N400 effects seen on articles and on nouns. The original DUK paper reports similar magnitude and topography of N400 effects on both the article and noun. It would be instructive to compare the location, timing, and effect size for cloze probability correlations between the article and the noun in the current, larger dataset. Additional tests for difference between the significant and null correlations would be valuable. For instance, is the reliable effect of graded cloze probability on nouns significantly different from the null effect of cloze probability on articles? This difference should be reliable if semantic integration is the key driver of N400 effects on nouns.

We already compare the effects of articles and nouns in our more powerful single trial-analysis, it shows a massive difference. We have also seen a single-trial re-analysis of the DUK data (by DUK), it shows the same pattern. We do not see the added value of the suggested analysis. We don’t think it is particularly instructive to know whether the correlations differ if this analysis is clearly problematic. Moreover, the noun-correlations are significantly different from zero, whereas the article-correlations go into the opposite direction at the channels where the noun-N400 effects are strong.

In a similar vein, what is the assumed model of the N400 generator? There are at least two published computational models that seek to explain the large body of N400 effects in the existing ERP literature. One proposes something akin to integration demands (Cheyette and Plaut, 2016), another proposes a form of (semantic) prediction error (Rabovsky and McRae, 2014). Such models provide a helpful context in which to discuss whether or not prediction violations at the level of form will lead to an N400 in the absence of any ongoing semantic integration demands. Ultimately, questions about the replicability or otherwise of the data presented by DUK are only informative to the extent that they are or, are not theoretically constraining. A discussion of these and other computational theories of the N400 are important for interpreting the present results, and how these results should be considered in the field of language comprehension going forward.

The two cited models are not models of sentence-level comprehension effects in N400 activity but of word-level comprehension, so it's difficult to say what they would predict on DUK type violations (if anything). This reviewer states that “these and other computational theories of the N400 are important for interpreting the present results”, but none of these models has ever said anything about ERP effects of pre-nominal manipulations in prediction-studies like DUK, and these studies have nothing or very little so say about form-prediction (Rabovsky et al., 2018), probably because they all involve purely semantic representations. It is not at all clear to us how these theories help us understanding whether the true population-level effect size is more like that reported by DUK or more like that reported here.

Cheyette and Plaut assume that the N400 ERP component reflects difficulty of semantic access, essentially the view-point of Kutas et al. Rabovsky and McRae argue that the N400 reflects semantic surprise, and a later model about sentence-level comprehension by Rabovsky, Hanssen and McClelland “treats N400 amplitudes as indexing the change induced by an incoming word in an implicit probabilistic representation of meaning”, and does not assume separate stages of access and integration. All these accounts assume the N400 component reflects a unitary process, which we think is unlikely, and, in fact, our own new results suggest that EEG activity in the N400 window is sensitive to different types of information at different times (Nieuwland et al., 2018). The only computational model that is accompanied by discussion of DUK-like effects is from Fitz and Chang (https://psyarxiv.com/frx2w/) which takes the N400 as a signal of error propagation. Fitz and Chang specifically discuss the fact that their model does not capture DUK-like N400 effects as a limitation of their model, and state that DUK-like effects would require a different set of mechanisms beyond their own model.

Other general comments for the authors to consider:

1) The authors conducted a single-trial analysis using linear mixed effects regression (LMER) analysis. This is the most appropriate analysis method since it includes both between-participant and between-item variance and combines all of the usable data from individual participants and single trials into one analysis. While LMER analysis is now preferred to correlation analysis used in the original study, it only came to prominence after the publication of the original paper. Pointing out the advantages of LMER analyses would enhance the current paper's impact. For instance, especially when analyzing effects of continuous predictor variables, LMER may offer better control over false-positives than conventional analyses. This is an important message for researchers studying the neuroscience of language, who generally do not undertake this approach.

We agree. In the Materials and methods section on our single-trial analysis, we added “Especially when analyzing effects of a continuous predictor variable such as cloze probability, LMER offers better control over false-positive results than the averaged-based correlation analysis of the original.” In the last paragraph of the Introduction, we added “and a single-trial analysis that modelled variance at the level of item and subject (with a linear mixed-effects model, which offers better control over false-positives than the replication analysis when analyzing effects of the continuous predictor cloze probability.”

2) The flip side of using single-trial LMER, however, is that it is not clear whether or not the link between cloze probability and neural responses is linear or logistic. The independent measure (cloze probability) used as a predictor variable is summarizing a binary outcome variable (i.e., did individual participants in the cloze test generate the article "a" or "an") as a probability. It is at least plausible that neural outcomes could be similarly binary: participants predict "a" or "an" and then don (or don't) generate a neural prediction error when an unexpected word is presented. Would reliable effects of cloze probability be observed with a logistic or binomial linking function? While this is not the form of graded prediction tested by DUK, it seems quite likely that averaging over many participants and many trials (as in the DUK correlation analysis) would generate a graded, linear relationship between cloze probability and neural outcomes, even if the underlying relationship was logistic or non-linear. This possibility should be considered, assessed and discussed.

To our knowledge, these analyses would require a binomial dependent variable, it is unclear how this would be applied to our data. But, regardless, this could be an interesting alternative analysis that others could use to explore our data (and later confirm its results in subsequent testing). We have discussed the issue of alternative analyses in our revised Discussion section.

3) Previous work has explored the presence/absence of fillers, which is another difference between the DUK study and the current one. While it is unlikely to explain the failure to replicate results, some additional discussion seems appropriate. Given the large scale of the present dataset there should be many opportunities for determining whether cloze probability effects (for nouns, if not for articles) are enhanced once participants have already encountered a number of distinctive, high/low cloze probability sentences.

We appreciate the interest in this issue of fillers. It is unclear what additional discussion this reviewer would like to see. We know that DeLong et al., also have run other experiments with the a/an manipulations with different kinds of fillers. Publication of those results will offer a better opportunity to determine the importance of fillers. In our own dataset, there are opportunities to examine effects of experiment-position, and, in fact, when we posted our BioRxiv pre-print about a year ago, Florian Jaeger already notified us of his intentions to perform such an analysis using the data we made available. Our data is available to anyone with an interest in pursuing such an exploratory analysis.

4) The fact that the nine labs inadvertently manipulated the timing of stimulus presentations of the critical articles and nouns (see footnote 3, Materials and methods section) is unlikely critical to their findings/conclusions. However, it would be useful to know whether the timing of early, visual form responses for written words that are delayed and non-delayed differ. Simply arguing that the change in timing is "unlikely to be noticed" and "after the N400 window" is making assumptions that can and should be tested with their ERP data.

We have rephrased the description in this footnote to make it more clear. First, as the original footnote stated, the timing was slightly off only in 4 labs, not all labs, as this reviewer writes. We have also analyzed only the data where the timing was as intended, which did not change the results, and we had mentioned this already in the footnote. Second, the articles were never themselves delayed, but the blank screen that followed them was 80 ms longer than intended. This is crucial, because it means that the first moment at which the timing is different from the intended timing is at 500 ms, which therefore cannot have impacted the article-N400s. This is what we meant by “after the N400 window”: this is a calculation, not an assumption. We have removed the references about slower reading times leading to more predictive processing to avoid further confusion.

5) The use of different baselines was confusing to multiple reviewers. A table that summarizes details of all the current analyses and how these correspond to the procedures used in the original DUK study would help the reader keep straight the different results.

There are only three different baseline procedures: none, 100ms or 500 ms. We used different baselines because DUK did not correctly report their procedure. We have explained and motivated our pre-registered baseline procedures and motivated changes to the baseline procedure in exploratory analyses. We have also changed “Original baseline procedure” in Figure 6 to avoid confusion, because this is ambiguous (it could mean what was reported or what was done).

All the procedures are summarized in the table 1. We do not think it is necessary to include it in the main text but will include it if the editor deems it important enough to include.

6) Both the original and current studies present a high percentage of a/an continuations in the cloze test. Could this procedure bias participants towards using a/an rather than "his", "hers", "their," etc. that aren't marked for the phonological form of the subsequent noun? After encountering several sentences in which the indefinite article is appropriate, participants may routinely complete the remaining sentences with an indefinite article continuation due to this sort of syntactic priming. A comment about this would be welcome.

Inspection of the raw cloze results suggests that this is a very unlikely scenario. We suspect that the original items were designed to elicit a/an responses instead of possessive pronouns, which would make sense given the goal of the experiment, but we cannot be certain, and, to our knowledge, the raw cloze data of DUK are not publicly available. In our own cloze data, there are two items where several participants use possessive pronouns (for the famous kite/airplane example item number 8, and another item with ‘her finger’ which is number 50 in the list. So, our participants did use possessive pronouns when those pronouns came to mind as best completions even more than halfway through the list.

7) In multiple places, the authors use phrasing about how the study argues for "a more limited role for prediction during language comprehension." This wording seems overly broad, conflating very different conceptions of prediction across different levels of representation. There is a deep theoretical division between proposed mechanisms of conceptual pre-activation (e.g., the concept of 'readiness' in the discourse literature, linked to the N400 literature beautifully in Jos Van Berkum's work) and mechanisms of predicting the form of the upcoming input (commitment to a particular linguistic formulation of the message). It seems likely that the former, long argued for in the discourse literature, might play a central role in language comprehension, while contributions from the prediction-of-form mechanism may be quite limited. It would be unfortunate if readers take the current study to cast doubt on both extremely different mechanisms, when it actually only speaks to the latter mechanism. Please consider rewording this throughout.

We have removed or reworded that phrase throughout the text.

Author details

1. Mante S Nieuwland

   1. Max Planck Institute for Psycholinguistics, Nijmegen, Netherlands
   2. School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Supervision, Validation, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   For correspondence

   mante.nieuwland@mpi.nl

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4001-6608

2. Stephen Politzer-Ahles

   1. Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University, Kowloon, Hong Kong
   2. Faculty of Linguistics, Philology and Phonetics, University of Oxford, Oxford, United Kingdom

   Contribution

   Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   For correspondence

   stephen.politzerahles@polyu.edu.hk

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-5474-7930

3. Evelien Heyselaar

   School of Psychology, University of Birmingham, Birmingham, United Kingdom

   Contribution

   Investigation, Writing—review and editing

   For correspondence

   evelien.heyselaar@mpi.nl

   Competing interests

   No competing interests declared

4. Katrien Segaert

   School of Psychology, University of Birmingham, Birmingham, United Kingdom

   Contribution

   Resources, Supervision, Writing—review and editing

   For correspondence

   k.segaert@bham.ac.uk

   Competing interests

   No competing interests declared

5. Emily Darley

   School of Experimental Psychology, University of Bristol, Bristol, United Kingdom

   Contribution

   Investigation

   For correspondence

   d12634@my.bristol.ac.uk

   Competing interests

   No competing interests declared

6. Nina Kazanina

   School of Experimental Psychology, University of Bristol, Bristol, United Kingdom

   Contribution

   Resources, Supervision, Writing—original draft, Writing—review and editing

   For correspondence

   nina.kazanina@bristol.ac.uk

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-7737-4279

7. Sarah Von Grebmer Zu Wolfsthurn

   School of Experimental Psychology, University of Bristol, Bristol, United Kingdom

   Contribution

   Investigation

   For correspondence

   sv13691@my.bristol.ac.uk

   Competing interests

   No competing interests declared

8. Federica Bartolozzi

   School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Investigation

   For correspondence

   Federica.Bartolozzi@mpi.nl

   Competing interests

   No competing interests declared

9. Vita Kogan

   School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, United Kingdom

   Contribution

   Investigation

   For correspondence

   vitavkogan@gmail.com

   Competing interests

   No competing interests declared

10. Aine Ito

    1. School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, United Kingdom
    2. Faculty of Linguistics, Philology and Phonetics, University of Oxford, Oxford, United Kingdom

    Contribution

    Investigation, Writing—review and editing

    For correspondence

    aineito@gmail.com

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0003-4408-8801

11. Diane Mézière

    School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, United Kingdom

    Contribution

    Investigation

    For correspondence

    d.c.meziere.1@student.rug.nl

    Competing interests

    No competing interests declared

12. Dale J Barr

    Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom

    Contribution

    Software, Formal analysis, Writing—review and editing

    For correspondence

    Dale.Barr@glasgow.ac.uk

    Competing interests

    No competing interests declared

13. Guillaume A Rousselet

    Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, United Kingdom

    Contribution

    Resources, Formal analysis, Supervision, Methodology, Writing—review and editing

    For correspondence

    guillaume.rousselet@glasgow.ac.uk

    Competing interests

    No competing interests declared

    "This ORCID iD identifies the author of this article:" 0000-0003-0006-8729

14. Heather J Ferguson

    School of Psychology, University of Kent, Canterbury, United Kingdom

    Contribution

    Resources, Supervision, Funding acquisition, Writing—review and editing

    For correspondence

    H.Ferguson@kent.ac.uk

    Competing interests

    No competing interests declared

15. Simon Busch-Moreno

    Division of Psychology and Language Sciences, University College London, London, United Kingdom

    Contribution

    Software, Formal analysis, Investigation

    For correspondence

    simon.busch.15@ucl.ac.uk

    Competing interests

    No competing interests declared

16. Xiao Fu

    Division of Psychology and Language Sciences, University College London, London, United Kingdom

    Contribution

    Investigation

    For correspondence

    xiao.fu.15@ucl.ac.uk

    Competing interests

    No competing interests declared

17. Jyrki Tuomainen

    Division of Psychology and Language Sciences, University College London, London, United Kingdom

    Contribution

    Resources, Supervision

    For correspondence

    j.tuomainen@ucl.ac.uk

    Competing interests

    No competing interests declared

18. Eugenia Kulakova

    Institute of Cognitive Neuroscience, University College London, London, United Kingdom

    Contribution

    Software, Formal analysis, Investigation

    For correspondence

    e.kulakova@ucl.ac.uk

    Competing interests

    No competing interests declared

19. E Matthew Husband

    Faculty of Linguistics, Philology and Phonetics, University of Oxford, Oxford, United Kingdom

    Contribution

    Resources, Supervision, Writing—review and editing

    For correspondence

    matthew.husband@ling-phil.ox.ac.uk

    Competing interests

    No competing interests declared

20. David I Donaldson

    Psychology, Faculty of Natural Sciences, University of Stirling, Stirling, United Kingdom

    Contribution

    Resources, Supervision, Writing—review and editing

    For correspondence

    d.i.donaldson@stir.ac.uk

    Competing interests

    No competing interests declared

21. Zdenko Kohút

    Department of Psychology, University of York, York, United Kingdom

    Contribution

    Investigation

    For correspondence

    zk578@york.ac.uk

    Competing interests

    No competing interests declared

22. Shirley-Ann Rueschemeyer

    Department of Psychology, University of York, York, United Kingdom

    Contribution

    Supervision, Writing—review and editing

    For correspondence

    shirley-ann.rueschemeyer@york.ac.uk

    Competing interests

    No competing interests declared

23. Falk Huettig

    Max Planck Institute for Psycholinguistics, Nijmegen, Netherlands

    Contribution

    Conceptualization, Funding acquisition, Writing—review and editing

    For correspondence

    falk.huettig@mpi.nl

    Competing interests

    No competing interests declared

• 8,398

  Page views

• 804

  Downloads

• 152

  Citations

Article citation count generated by polling the highest count across the following sources: Scopus, Crossref, PubMed Central.