bioRxiv2017
In the last few decades, the idea that people routinely and implicitly predict upcoming words during language comprehension has turned from a controversial hypothesis to a widely-accepted assumption. Current theories of language comprehension1-3 posit prediction, or context-based pre-activation, as an essential mechanism occurring at all levels of linguistic representation (semantic, morpho-syntactic and phonological/orthographic) and facilitating the integration of words into the unfolding discourse representation. The strongest evidence to date for phonological pre-activation comes from DeLong, Urbach and Kutas4, who monitored participants electrophysiological brain responses as they read sentences, presented one word at a time, with expected/unexpected indefinite article + noun combinations like, \"The day was breezy so the boy went outside to fly a kite/an airplane\". The sentences varied expectations ( cloze probability) for a consonant- or vowel-initial noun, as determined in a sentence-completion task using other participants. Expectedly, the amplitude of the N400 event-related potential (ERP) decreased (became less negative) with increasing cloze reflecting ease of processing5-6. Whereas the decreased N400 at the noun could be due to its pre-activation or because high-cloze nouns are easier to integrate, crucially, N400s at the immediately-preceding article a or an showed the same relationship with cloze, i.e., encountering an indefinite article that mismatched a highly-expected word (e.g., an when expecting kite) also elicited a larger N400. This led to the claim that participants pre-activated highly-expected nouns, including their initial phonemes, based on the preceding context, with larger N400s on mismatching articles reflecting disconfirmation of this prediction.\n\nThe Delong et al. study warranted stronger conclusions than related results available at the time. Unlike previous work, it did not rely on the precursory visual-depiction of upcoming nouns, clearly de-confounded prediction and integration effects, and tested for graded phonological pre-activation of specific word form. Correspondingly, the study has been enthusiastically received as strong evidence for probabilistic phonological pre-activation, receiving over 650 citations to date and featuring in authoritative reviews2-3. However, there is good cause to question the soundness of the original finding (and the appropriateness of the analysis used). Attempts to replicate the critical article-effect have failed7. Moreover, an earlier, alternative analysis of the same data by the authors8 failed to reach statistical significance, but was omitted from the published report.\n\nTo obtain more definitive evidence, we conducted a direct replication study spanning 9 laboratories (Ntotal = 334). We pre-registered one replication analysis that was faithful to the original, and one single-trial analysis that modeled subject- and item-level variance using linear mixed-effects models. Applying the replication analysis to our article data (Figure 1a), the original finding did not replicate: no laboratory observed a significant negative relationship between cloze and N400 at central-parietal electrodes. In contrast, the negative relationship was successfully replicated for the nouns: 6 laboratories observed such an effect and 2 laboratories observed relatively strong but non-significant effects in the expected direction (range r = .30 to .50). In the single-trial analysis (Fig. 1b-c), there was no statistically significant effect of cloze on article-N400s, also with stricter control for pre-article voltage levels (Supplementary Fig. 1). Crucially, there was a strong and significant cloze effect on noun-N400s (in all laboratories), which was significantly different from that on article-N400s. We observed no significant differences between laboratories for article or noun effects. Exploratory Bayesian analyses with priors based on DeLong et al. further support our conclusions (Fig. 1d, Supplementary Fig. 2). Finally, a control experiment confirmed our participants sensitivity to the a/an rule during online language comprehension (Supplementary Fig. 3).\n\nO_FIG O_LINKSMALLFIG WIDTH=173 HEIGHT=200 SRC=\"FIGDIR/small/111807_fig1.gif\" ALT=\"Figure 1\">\nView larger version (72K):\norg.highwire.dtl.DTLVardef@1231330org.highwire.dtl.DTLVardef@1c0eb43org.highwire.dtl.DTLVardef@95a28forg.highwire.dtl.DTLVardef@1e38ee6_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOFigure 1C_FLOATNO A multi-lab failure to replicate evidence for probabilistic pre-activation of phonology. (a) Pre-registered replication analysis: Pearsons r correlations between ERP amplitude and article/noun cloze probability per EEG channel (* P < 0.05) and per laboratory. (b, c) Pre-registered single-trial analysis: (b) Grand-average ERPs elicited by relatively expected and unexpected words (cloze higher/lower than 50%) at electrode Cz, with standard deviation are shown in dotted lines, and (c) the relationship between cloze and N400 amplitude as illustrated by the mean ERP values per cloze value (number of observations reflected in circle size), along with the regression line and 95% confidence interval. A change in article cloze from 0 to 100 is associated with a change in amplitude of 0.296 {micro}V (95% confidence interval: -.08 to .67), {chi}2(1) = 2.31, p = .13. A change in noun-cloze from 0 to 100 is associated with a change in amplitude of 2.22 {micro}V (95% confidence interval: 1.75 to 2.69), {chi}2(1) = 56.5, p < .001. The effect of cloze on noun-N400s was statistically different from its effect on article-N400s, {chi}2(1) = 31.38, p < .001. (d) Bayes factor analysis associated with the replication analysis, quantifying the obtained evidence for the null hypothesis (H0) that N400 is not impacted by cloze, or for the alternative hypothesis (H1) that N400 is impacted by cloze with the size and direction of effect reported by DeLong et al. Scalp maps show the common logarithm of the replication Bayes factor for each electrode, capped at log(100) for presentation purposes. Electrodes that yielded at least moderate evidence for or against the null hypothesis (Bayes factor of [≥] 3) are marked by an asterisk. At posterior electrodes where DeLong et al. found their effects, our article data yielded strong to extremely strong evidence for the null hypothesis, whereas our noun data yielded extremely strong evidence for the alternative hypothesis (upper graphs). These results were also found when applying a 500 ms pre-word baseline correction (lower graphs).\n\nC_FIG O_FIG O_LINKSMALLFIG WIDTH=198 HEIGHT=200 SRC=\"FIGDIR/small/111807_figS1.gif\" ALT=\"Figure 1\">\nView larger version (32K):\norg.highwire.dtl.DTLVardef@bad459org.highwire.dtl.DTLVardef@1cb4d98org.highwire.dtl.DTLVardef@534676org.highwire.dtl.DTLVardef@1371c54_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOSupplementary Figure 1.C_FLOATNO Exploratory single-trial analyses: The relationship between cloze and ERP amplitude as illustrated by the mean ERP values per cloze value (number of observations reflected in circle size), along with the regression line and 95% confidence interval, from four exploratory analyses. We performed tests which used longer baseline time windows (200 ms, upper left panel; 500 ms, upper right panel) to better control for pre-article voltage levels, or which used the pre-registered baseline and applied a 0.1 Hz high-pass filter (lower left panel) to better control for slow signal drift (while presumably not affecting N400 activity). All three tests reduced the initially observed effect of article-cloze (200 ms baseline, = .25, CI [-.12, .62], {chi}2(1) = 1.35, p = .19; 500 ms baseline, = .14, CI [-.25, .53], {chi}2(1) = 0.46, p = .50; 0.1 Hz filter: = 0.09, CI [-.22, .41], {chi}2(1) = 0.33, p = .56). An analysis in the 500 to 100 ms time window before article-onset (lower right panel) revealed a non-significant effect of cloze that resembled the pattern observed after article-onset, = .16, CI [-.07, .39], {chi}2(1) = 1.82, p = .18. Combined, these results suggest that the results obtained with the pre-registered analysis at least partly reflected the effects of slow signal drift that existed before the articles were presented.\n\nC_FIG O_FIG O_LINKSMALLFIG WIDTH=161 HEIGHT=200 SRC=\"FIGDIR/small/111807_figS2.gif\" ALT=\"Figure 2\">\nView larger version (35K):\norg.highwire.dtl.DTLVardef@124ffa2org.highwire.dtl.DTLVardef@af29aorg.highwire.dtl.DTLVardef@bd69d1org.highwire.dtl.DTLVardef@16e3b98_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOSupplementary Figure 2.C_FLOATNO Results from exploratory Bayesian mixed-effects model analyses, represented by posterior distributions for the effect of cloze on ERP amplitudes in the N400 window. The x-axis shows cloze effect sizes (i.e., changes in microvolts associated with an increase from 0% cloze probability to 100% cloze probability). The black line indicates the posterior distribution of effects; higher values of the posterior density at a given effect size indicate higher probability that this is the true effect size in the population. The peak of the posterior distribution roughly corresponds to the point estimate of the effect size (the regression coefficient) fitted from the Bayesian mixed effect model, i.e., the most likely value of the true effect size. The middle 95% of the posterior distribution, shaded in pink, corresponds to a two-tailed 95% credible interval for the effect sizei.e., an interval that we can be 95% confident contains the true effect. The green dotted line indicates the prior distribution (i.e., our expectation about where the true effect would lie before the data were collected), which is centered on 1.25V, the effect observed by Delong and colleagues (2005). The black connected dots illustrate the ratio between the posterior and prior distribution (i.e., the Bayes Factor) at the effect size of 0V; for example, a Bayes Factor of 4 suggests we can be 4 times more certain that the true effect is zero after having conducted this experiment than before, or, in other words, that the data increased our confidence in the null effect of zero fourfold. We performed these analyses for each of the linear mixed-effects model analysis we performed. We note that in all the article-analyses, the posterior probability of the estimated effect being greater than zero is around 80 or 90%, but this is also the case for the pre-stimulus variable, suggesting that the observed patterns arise before the articles are seen. In none of our article-analyses did zero lie outside the obtained credible interval, whereas for the nouns, zero lay outside the credible interval. These results are consistent with a failure to replicate the DeLong et al. article-effect and successful replication of the noun-effect.\n\nC_FIG O_FIG O_LINKSMALLFIG WIDTH=133 HEIGHT=200 SRC=\"FIGDIR/small/111807_figS3.gif\" ALT=\"Figure 3\">\nView larger version (25K):\norg.highwire.dtl.DTLVardef@ab14forg.highwire.dtl.DTLVardef@1fed126org.highwire.dtl.DTLVardef@55369corg.highwire.dtl.DTLVardef@747668_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOSupplementary Figure 3.C_FLOATNO P600 effects at electrode Pz per lab associated with flouting of the English a/an rule in the control experiment. Plotted ERPs show the grand-average difference waveform and standard deviation for ERPs elicited by ungrammatical expressions (an kite) minus those elicited by grammatical expressions (a kite). This control experiment followed in the same experimental session as the main experiment and was carried to rule out that an observed lack of a statistically significant, article-elicited prediction effect in the main experiment reflected a general insensitivity of our participants to the a/an rule. In each laboratory, nouns following incorrect articles elicited a late positive-going waveform compared to nouns following correct articles, starting at about 500 ms after word onset and strongest at parietal electrodes. This standard P600 effect was confirmed in a single-trial analysis, {chi}2(1) = 83.09, p < .001, and did not significantly differ between labs, {chi}2(8) = 8.98, p = .35.\n\nC_FIG\n\nDespite a sample size 10 times larger than the original and improved statistical analysis, we observed no statistically significant effect of cloze on article-N400s, while replicating the strong and statistically significant effect of cloze on noun-N400s4,6. The effect of cloze on article-N400s, if existent, must be very small to evade detection given our expansive approach. Whether such an effect would constitute convincing evidence for routine phonological pre-activation as assumed in theories of language comprehension3 can be questioned, but, more generally, such an effect cannot be meaningfully studied in typical small-scale studies. Consequently, current theoretical positions may be based on potentially unreliable findings and require revision. In particular, the strong prediction view that claims that pre-activation routinely occurs across all - including phonological - levels3, can no longer be viewed as having strong empirical support.\n\nOur results do not constitute evidence against prediction in general. We note a lack of convincing evidence specifically for phonological pre-activation, which would have to be measured before a noun appears and unobscured by processes instigated by the noun itself.\n\nHowever, our results neither support nor necessarily exclude phonological pre-activation. Unlike gender-marked articles9 (e.g., in Dutch or Spanish) that agree with nouns irrespective of intervening words, English a/an articles index the subsequent word, which is not always a noun. Maybe our participants did not use mismatching articles to disconfirm predicted nouns, possibly because it was not a viable strategy (American and British English corpus data show a mere 33% chance that a noun follows such articles). Perhaps a revision of the predicted meaning is required to trigger differential ERPs.\n\nDeLong et al. recently described filler-sentences in their experiment10, cf. 7, which were omitted from their original report, and were neither provided nor mentioned to us upon our request for their stimuli. DeLong used the existence of these filler-sentences to dismiss an alternative explanation of their results, namely that an unusual experimental context wherein every sentence contains an article-noun combination leads participants to strategically predict upcoming nouns. Importantly, we failed to replicate their article-effects despite an experimental context that could inadvertently encourage strategic prediction. Therefore, the difference between their experiment and ours cannot explain the different results, and may even strengthen our conclusions.\n\nIn sum, our findings do not support a strong prediction view involving routine and probabilistic pre-activation of phonological word form based on preceding context.\n\nMoreover, our results further highlight the importance of direct replication, large sample size studies, transparent reporting and of pre-registration to advance reproducibility and replicability in the neurosciences.