Experimental paradigm for auditory-visual label learning.

A) Subjects were exposed to four different visual-auditory pairs during three days (6 repetitions of each pair, 3 minute video). Two pairs were always presented in the ‘visual-then-auditory’ order (object to label), and two in the ‘auditory-then-visual’ (label to object) order. During the test phase, this canonical order was kept on 80% of trials, including 10% of incongruent pairs to test memory of the learned pairs, and was reversed on 20% of the trials. On reversed trials, half the pairs were congruent and half were incongruent (each 10% of total trials), thus testing reversibility of the pairings without affording additional learning. B,C) Activation in sensory cortices. Although each trial comprises auditory and visual stimuli, these could be separated by the temporal offsets. Images show significantly activated regions in the contrasts image > sound (red-yellow) and sound > image (blue-light blue), averaged across all subjects and runs for humans (B) and monkeys (C). D,E) Average finite-impulse-response (FIR) estimate of the deconvolved hemodynamic responses for humans (D) and monkeys (E) within clusters shown in B and C respectively, separately for visual-audio (VA) and audio- visual (AV) trials. Sign flipped on y-axis for monkey responses.

Congruity effects in the auditory-visual task in humans (experiment 1).

A) areas activated by incongruent trials more than by congruent trial in canonical trials (red), reversed trials (blue), and their overlap (green). Brain maps are thresholded at pvoxel < 0.001& pcluster < 0.05 corrected for multiple comparisons across the brain volume. No interaction effect was observed between congruity and canonicity. B) Average FIR estimate of the deconvolved hemodynamic responses within significant clusters in the left hemisphere, separately for VA and AV trials.

Congruity effect in Experiment 1 in humans (n=31).

Congruity effects in the auditory-visual task in monkeys (experiment 1).

A) significant clusters from the incongruent-congruent canonical contrast. No significant clusters were found for the reversed direction. B) significant clusters from the interaction between congruity and canonicity. (pvoxel<0.001 & pcluster<0.05 for both maps) C,D) Average FIR estimate of the deconvolved MION responses within the clusters from the incongruent-congruent canonical contrast, averaged over VA and AV trials. All clusters in early visual areas were taken together to create figure C. Average of 2 animals.

Congruity effect in Experiment 1 in monkeys (n=2)

Visual-visual label learning in humans and monkeys (experiment 2).

A, Experiment paradigm. Subjects were habituated to 4 different visual-visual pairs during three days. Two pairs were in the ‘object-then-label’ order and two pairs in the ‘label-then-object’ order. For the monkeys, one object in each direction was associated with a high reward while the other one was associated with a low reward, making reward size orthogonal to congruity and canonicity (See Supplementary Figure 2 for details). B, monkey fMRI results. Significant clusters (pvoxel<0.001 & cluster volume >50) from the incongruent- congruent canonical contrast (left) and the interaction between congruity and canonicity (right). C, human fMRI results. Areas more activated by incongruent trials more than by congruent trial in canonical trials (red), reversed trials (blue), and their overlap (green) (right) (pvoxel<0.005 & cluster volume >50). No red voxels are visible because all of them figure in the overlap (green). D, Human behavioral results. After learning, human adults rated the familiarity of different types of pairs (including a fifth category of novel, never seen pairings). Each dot represents the mean response of one subject in each condition. Although the reversed congruent trials constituted only 10% of the trials, they were considered almost as familiar as the canonical congruent pairs.

Congruity effect in Experiment 2 in monkeys (n=3)

Congruity effect in Experiment 2 in humans (n=23)

ROIs analyses: F-values of ANOVAs performed on the averaged betas of the main task across the 10% best voxels selected in an independent localizer in ROIs commonly activated in language and mathematical tasks. The language ROIs are presented as red areas on the sagittal (x=-50) and coronal (y=- 58) brain slices and the mathematical ROIs as yellow areas. The left white area corresponds to the VWFA; n=52; df=50; pFDRcor: *** <0.001, ** < 0.01, *< 0.05, ° < 0.1.

Summary of the two experiments in humans and monkeys.

(In experiment 1, pvoxel < 0.001 & pcluster < 0.05 for humans and monkeys. In experiment 2, pvoxel<0.005 & cluster volume >50 in humans and pvoxel<0.001 & cluster volume >50 in monkeys.)

Stimulus sets for experiment 1.

A) Complete description of the task paradigm for visual-visual label learning. Subjects were habituated to 4 different visual-visual pairs during three days. Two pairs were in the ‘object-label’ order and two pairs in the ‘label-object’ order. During the test phase, the same canonical order was kept in 80% of the trials, including 10% of incongruent pairs. In reversed trials (20% of trials), the pairs were either congruent (10%) or incongruent (10%) with the learning. For the monkeys, one pair in each direction was associated with a high reward while the other one was associated with a low reward, making the reward size orthogonal to congruity and canonicity. B) Stimulus sets for experiment 2 in monkeys. Humans were tested with stimulus set 2.

Effect of reward for the visual-visual task in non-human primates.

A) Significant clusters from the incongruent-congruent canonical contrast in low reward trials. B) Significant from the incongruent-congruent canonical contrast in high reward trials. C) Significant clusters from the interaction between congruity and reward. pvoxel<0.001 & pcluster <0.05 in all panels.

Analyses of all human participants in experiments 1 and 2 merged.

A) Main effect of experiment. B) Main effect of congruity, C) Effect of congruity in the canonical trials and D) in the reversed trials. E) No significant cluster was observed for the interaction canonicity X congruity. F) slices in the 3 planes showing the only significant cluster in the Experiment X Congruity interaction. pvoxel<0.001 & pcluster <0.05 in all panels