Overview of the study methods.

fMRI data from subjects playing Super Mario Bros (a) is processed and reduced to time series with a brain parcellation (b). Artificial models are trained with different objectives and tested for their ability to predict brain activity, using different objectives during training (c): reinforcement learning, behavioural cloning and visual supervised learning (ResNet variants). Brain encoding performance (d) is evaluated both within-distribution (on the game levels used for training) and out-of-distribution (on game levels not seen during training), highlighting each model’s ability to generalize beyond their training data (e).

Comparison of the brain encoding R² scores of the different models.

a Brain encoding R2 scores, averaged over brain parcels. Each dot corresponds to the value for a layer and a subject (the ResNet being evaluated on just one layer, it presents less sample points). b Mean brain encoding R² score per layer for each model. Each plot represents the scores averaged of the whole brain (top left) or different ROIs of the visual cortex. c Brain encoding maps per model. The maps represent the brain encoding R² score per parcel, averaged over subjects, for each model’s conv3 layer (except for the ResNet). For each model we see a similar spatial distribution of the R² score. The regions that get the highest R² scores are located in the visual cortex, the dorsal attentional path and in the motor cortex. d Brain encoding maps per subject. The maps represent the brain encoding R² score averaged over models, for the conv3 layer (except for the ResNet). The maps of the different subjects show more noticeable differences in R² amplitude than the maps per model, but the overall spatial distributions remain very similar. The regions that get the highest R² scores are located in the visual cortex, the dorsal attentional path and in the motor cortex.

Training effects on brain encoding and task performance.

a: Brain encoding R² for each model and subject across training checkpoints (averaged over brain parcels, evaluated on the test set). b: Difference maps showing changes in R² between early training (10–15%) and the final checkpoint, averaged across subjects. c: Correlation between brain encoding and game score for the PPO model across checkpoints. Dashed lines show linear fits. d: Correlation between brain encoding and imitation score for the Imitation models across checkpoints. Dashed lines show linear fits.

Difference of brain encoding R² between models for each subject and layer.

The R² scores are averaged across brain parcels and across sequences of 100 TRs. The asterisks denote significance of a two sided Wilcoxon signed-rank test. One, two or three asterisks correspond respectively to a p-value < 0.05, a p-value FDR corrected (Benjamini/Yekutieli) < 0.05 and a p-value Bonferroni corrected < 0.05.

Cross-level correlations of brain encoding and task performance.

a: PPO brain encoding versus game score across levels, shown separately for each subject. b: Imitation model brain encoding versus imitation score across levels, by subject. c: PPO game score versus imitation model score across levels, by subject. d: PPO and imitation model brain encoding across levels, showing near-perfect correlations.

Out-of-distribution evaluation.

a Behavioral scores comparison between the average on within-distribution levels (red) and the scores and each of the OOD levels (orange and blue). For the PPO, the behavioral score is the game performance score. For the Imitation models, the score is the imitation score. b Brain encoding maps of the OOD levels, for each model (conv3 layer for all models but ResNet) and averaged over subjects. c Visual and dorsal attentional ROIs from the Yeo7 atlas. d Brain encoding map for the within-distribution levels, averaged over subjects and models. e Evolution of the brain encoding accuracy across training in the visual and dorsal attentional ROIs.