The 16 predefined line segments of McClelland and Rumelhart (1981) and five example letters

Summary of the MEG results obtained by Vartiainen et al. (2011).

A: Examples of stimuli used in the MEG experiment. Each stimulus contained 7–8 letters or symbols. B: Source estimate of the evoked MEG activity, using minimum norm estimate, dynamic statistical parametric mapping (MNE-dSPM). The grand-average activity to word stimuli, averaged for three time intervals, is shown in orange hues. For each time interval, white circles indicate the location of the most representative left-hemisphere ECD for each participant, as determined by Vartiainen et al. (2011). C: Grand-average time course of signal strength for each group of ECDs in response to the different stimulus types. The traces are color-coded to indicate the stimulus type as shown in A. Shaded regions indicate time periods over which statistical analysis was performed. D: For each group of ECDs shown in B, and separately for each stimulus type (different colors, see A), the distribution (and mean) of the grand-average response amplitudes to the different stimulus types, obtained by integrating the ECD signal strength over the time intervals highlighted in C. Whenever there is a significant difference (linear mixed effects (LME) model, p < 0.05, false discovery rate (FDR) corrected) between two adjacent distributions, the corresponding difference in means is shown.

Overview of the proposed computation model of feed-forward processing during visual word recognition.

A: The VGG-11 model architecture, consisting of five convolution layers, two fully connected layers and one output layer. B: Examples of the images used to train the model.

building a model that can simulate the type-I, type-II and N400m responses.

Starting from a VGG-11 model, we made adjustments to the base architecture and training diet of the model to produce variations which simulated activity better matches the response profiles of the three MEG evoked components. A: For each layer, the response profile, i.e. the z-scored magnitude of ReLU activations in response to the same stimuli as used in the MEG experiment, is shown. Whenever there is a significant difference (t-test, p < 0.05, FDR corrected) between two adjacent distributions, the corresponding difference in means is shown. Layers for which the response pattern was qualitatively similar to that of the type-I, type-II or N400m component are outlined with a box of the appropriate color. B: Correlation between the layers of each model (horizontal axis) and the three MEG evoked components (different curves). Layers for which the response profile (A) was judged to qualitatively correspond to one of the MEG components (Figure 2D) are indicated as filled squares. Noise ceilings for the MEG components are drawn as horizontal lines.

Exploring changes in model architecture.

Model variations were constructed with a different number of convolution and fully connected layers to see what architecture produces activity that is the most like the three MEG components. A: For each layer, the response profile, i.e. the z-scored magnitude of ReLU activations in response to the same stimuli as used in the MEG experiment, is shown. Whenever there is a significant difference (t-test, p < 0.05, FDR corrected) between two adjacent distributions, the corresponding difference in means is shown. Layers for which the response pattern was qualitatively similar to that of the type-I, type-II or N400m component are outlined with a box of the appropriate color. A black line separates convolution layers from fully connected layers. B: Correlation between the layers of each model (horizontal axis) and the three MEG evoked components (different curves). Layers for which the response profile (A) was judged to qualitatively correspond to one of the MEG components (Figure 2D) are indicated as filled squares. Noise ceilings for the MEG components are drawn as horizontal lines.

Impact of several hyperpa-rameters on the correlation between model and brain.

This figure shows the correlation between the response profiles of the type-I, type-II and N400m evoked MEG components and the three layers in the models whose response profiles best match each of the MEG components. Estimated noise ceilings for each of the MEG components are shown as vertical lines. Each panel shows the impact of tweaking a hyper-parameter and has an illustration to indicate the property of the model affected by the hyperparameter. The settings of the hyperparameters chosen for the final modal are highlighted in grey.

A closer look at the relationship between the final model and MEG responses.

A: A closer look at the relationship between the response profiles of the MEG responses and three layers of the model that qualitatively best capture those MEG responses. Kernel density distributions are shown at the borders. B: Correlation between the MNE-dSPM source estimate and the model. Grand-average source estimates were obtained in response to each stimulus. The correlation map was obtained by correlating the activity at each source point with that for the chosen three layers of the model. The correlation map is shown at the time of peak correlation (within the time windows indicated in Figure 2C). Only positive correlations are shown.

Post-hoc exploration of various experimental contrasts.

For each contrast, four sample stimuli are shown to demonstrate the effect of the manipulated stimulus property and below are the correlation between the manipulation and the amount of activity in each layer of the final model. For the experimental contrasts indicated with a number, one or more confounding factors were corrected for (partial correlation). Different colors indicate convolution layers (blue), fully connected layers (orange) and the output layer (green).