Multisensory learning enhances memory performance.

a. Left, Apparatus for multisensory training and testing. Right, experimental timeline. b. Protocols. Green and blue squares represent colours, light and dark gray squares represent OCT and MCH odours. Visual (V) learning: colours used as CS+ and CS−. Olfactory (O) learning: odours used as CS+ and CS−. Congruent (C) protocol: colours+odours combined as CS+ and CS−. Same colour+odour combinations used for training and testing. Incongruent (I) protocol: colours+odours combined as CS+ and CS− but combinations were switched between training and testing. Olfactory retrieval (OR): colour+odour combinations used for training but only odours for testing. Visual retrieval (VR): colour+odour combinations used for training but only colours for testing. c and d. Top, training and testing timelines. Bottom, immediate (c) and 6 h (d) memory for (V), (O), (C) and (I) protocols. e and f. Top, timelines. Bottom, multisensory training with colours+odours tested immediatly (e) and 6 h (f) after training for each individual modality. Asterisks denote significant difference (P < 0.05). Data presented as mean ± standard error of mean (SEM). Individual data points displayed as dots correspond to independent experiments. Groups compared using one-way ANOVA with Tukey’s test (c, d) and unpaired two-sided t-test (e,f), exact P values and comparisons are given in Supplementary Table 1. N values for each experiment are: c, n=8 for V and n=10 for O, C and I; d, n=10; e, n=10; f, n=14 for O and OR and n=10 for V and VR.

Enhanced performance following multisensory learning requires visually-responsive γd and αβp Kenyon Cells.

a. Top left, schematic of γ dorsal [γd] KCs. Bottom left, timeline with temperature shifting (dashed line). Right, blocking output of γd KCs during testing using MB607B-GAL4; UAS-Shits1 in the congruent protocol. b. Blocking γd KC output during testing in the incongruent protocol. c. Top left, schematic of αβp KCs. Bottom left, timeline with temperature shifting. Right, blocking αβp KC output during testing using c708a-GAL4; UAS-Shits1 in the congruent protocol. d. Blocking αβp KC output during in the incongruent protocol. e and f. Top, timeline with temperature shifting. Bottom, blocking γd (e) and αβp (f) KCs output during testing of Olfactory Retrieval of multisensory memory. Asterisks denote significant difference (P < 0.05). Data presented as mean±SEM. Individual data points displayed as dots correspond to independent experiments. All groups compared using one-way ANOVA with Tukey’s test, exact P values and comparisons are given in Supplementary Table 1. N values for each experiment are: a, b, d, f, n=12; c, e, n=10.

DPM and APL connectivity to KCs allows for multisensory stimulus binding.

a. Left, 3D representation of MB (light gray) with DPM neuron γ lobe neurites (teal). DPM trifurcates (asterisk) into dorsal (dark teal), ventral (light teal) and γ1 compartment (teal) branches. Neurites shaded by Strahler order and twigs with Strahler order <1 were pruned. Right, detail of the γ lobe with γ1-γ5 compartment borders (dashed lines). DPM presynapses to γd KCs (yellow spheres) co-localize on dorsal branch in γ5. Input synapses from γm KCs (gray spheres) to DPM locate throughout ventral and dorsal branches. In γ5, 450 of 585 γm KCs made synapses with the DPM dorsal branch, where 89 of 98 γd KCs also receive DPM input. APL neuron inputs (magenta spheres) localize along both DPM branches. b. 2D dendrogram projection of DPM neurites (shades of teal). The dorsal branch of the horizontal lobe’s γ2-γ5 compartments are dark teal, the rest of the γ lobe projections are mid teal and those in the other lobes are lighter teal. Note: all but γ lobe neurites are downsized for visibility. Neurites with Strahler order <1 are pruned and connectivity is shown accordingly. Projections in the γ1 compartment are marked and split in the γ2-5 compartments into dorsal and ventral branches. Marking is according to DAN connectivity (shaded areas). Synapses (spheres) are only marked in the γ lobe compartments. γ1 compartment is marked and γ2-5 are split between dorsal and ventral DPM neuron branches. Inputs from γm KCs (gray spheres) and outputs to γd KCs (yellow spheres) co-localize on compartment-specific branches. Inhibitory inputs from APL (magenta spheres) are distributed across DPM neurites. c. A 2-dimensional dendrogram projection of DPM neuron neurites as in b but with the reverse polarity of inputs from γd and KCs and γm KCs shown. Inputs from γd KCs (yellow spheres) and outputs to γm KCs (gray spheres) also colocalize on compartment specific branches of the DPM neurons. APL inputs (magenta) are distributed across both dorsal and ventral branches and concentrated around the branching point at the base of the vertical lobes in the α’1 compartment (asterisk). Note: Some putative parts of the dorsal γ lobe branch could not be allocated to a compartment due to lack of DAN input. One branch that bears γ KC synapses (hashtag) was identified as the β’ lobe branch that enters the MB close to the midline. It is likely that other DPM neurite tips that have γ KC synapses are artifacts where healing has merged free-floating γ lobe branchlets to DPM neurites innervating the other lobes. d. Left, 3D representation of MB (light gray) with APL neuron γ lobe neurites (purple). Neurites shaded by Strahler order and twigs with Strahler order <1 were pruned. A 2-dimensional dendrogram projection of APL neuron neurites (magenta). Note: Neurites with Strahler order <1 were pruned and connectivity is shown accordingly. Compare with (a). PPL1-γ1pedc and PAM-γ5 DAN input synapses (red and green spheres respectively) are marked in the γ1 and γ5 compartments (shaded areas). Synapses to DPM neurons (teal spheres) colocalize with DAN inputs in the γ lobe.

DPM and APL neurons are required for multisensory memories.

a. Top, Anatomy of APL neuron. Training and testing timelines with constant restrictive temperature (dashed line). Bottom, testing 6 h Olfactory Retrieval memory performance after (a) multisensory appetitive training, and (b) unisensory Olfactory learning in flies with adult-restricted RNAi knockdown of Dop2R in APL neurons using tubPGAL80ts; VT43924-GAL4.2. c. Top, DPM neuron schematic. Timeline with temperature shifting. Bottom, blocking DPM neuron output with VT64246-GAL4; UAS-Shits1 during training (c) or testing (d) 6 h Olfactory Retrieval performance. e-g. Top left, schematic of γ KCs. Top, training and testing timeline with temperature shifting (dashed line). Bottom, blocking output of γ KCs during multisensory training using MB009B-GAL4; UAS-Shits1 and testing 6 h Olfactory Retrieval (e) or Visual Retrieval (f), or (g) during olfactory training and testing 6 h olfactory memory performance. h. Timeline. RNAi knockdown of 5-HT2A, 5-HT2B or 5-HT7 receptors in γd KCs with MB607B-GAL4 and testing of Olfactory Retrieval 6 h after multisensory training. Data represented as mean±SEM. Individual data points displayed as dots. Asterisks denote significant difference (P < 0.05). Groups compared using one-way ANOVA with Tukey’s test (a-h), exact P values and comparisons are given in Supplementary Table 1. N values for each experiment are: a, b, n=12; c, n=8 for Shi and n=9 for other groups; d, n=10; e-g, n=8 flies; h, n=10. i. Model of DPM microcircuit bridging of odour and colour specific KCs following multisensory learning.

Engram expansion benefits new learning

a. MB model for appetitive multisensory colour+odour training followed by unisensory odour or colour testing. γm KCs receive dendritic olfactory input and γd visual input. Both γKC types project axons through γ1-γ5 compartments of the MB γ lobe. Appetitive training (left) engages reward DANs (green) innervating γ4 and γ5, whose released dopamine encodes learning by depressing synapses49 between odour-activated KCs and avoidance-directing Mushroom Body Output Neurons (MBONs; not illustrated)14. Dopamine signalling during multisensory learning also binds γm and γd KC activity in γ4-5 compartments. During future unisensory odour testing (middle), odour excites specific γm KCs (thick gray), which in turn activate γd axons in γ4-5 compartments (gray dashed lines to yellow). Reverse γd-mediated activation of γm KCs occurs with unisensory colour testing (right). b. MB model for aversive multisensory training followed by odour testing. Aversive multisensory training (left) engages punishment DANs (red) that depress synapses16,17 between γm and γd KCs and approach-directing γ1 and γ2 MBONs16,17 (not shown) while also binding γd and γm KC activity in these compartments. Unisensory odour testing (right) excites specific γm KCs which activate γd axons from γ1 forward. c and d. Prior aversive multisensory learning enhances future appetitive but not aversive odour learning. Left, protocols. Starved flies were divided into: Group I, aversive multisensory training, and Group II, aversive unisensory odour training. 3 h later both groups were trained with odours and sugar reward (using the same CS+/CS− odours as for initial training) and tested immediately afterwards. Right, memory performance. c. Group I initially trained with multisensory aversive protocol performed better than Group II initially aversively trained with only odours. d. Group I initially trained with multisensory appetitive protocol did not outperform Group II initially trained with only odours. Asterisks denote significant differences (P < 0.05). Data presented as mean±SEM. Individual data points displayed as dots and correspond to independent experiments. Groups compared using unpaired two-sided t-test (c,d), exact P values and comparisons are given in Supplementary Table 1. N values for each experiment are: c, n=12; d, n=8.

Memory performance is most robust when colour and odour combinations are consistent during acquisition and retrieval.

a. Top left, multisensory protocols. Green and blue squares represent colours, light and dark gray squares represent OCT and MCH odours. Visual (V) learning: colours used as CS+ and CS−. Olfactory (O) learning: odours used as CS+ and CS−. Congruent (C) protocol: colours+odours were combined as CS+ and CS−. Same colour+odour combinations used during training and testing. Incongruent (I) protocol: colours+odours were combined as CS+ and CS− but combinations were switched between training and testing. Bottom left, training and testing timeline. Right, 24 h memory performance for V, O, C and I protocols. Combining colours+odours in the congruent (C) protocol enhanced 24 h performance, compared to that obtained with unisensory V or O learning. Incongruent (I) pairing of colours and odours abolished the multisensory enhancement of 24 h memory. B. Left, training and testing timeline. Middle, multisensory protocols. Right, immediate memory performance. Flies showed a significantly higher memory following the C than the I protocol. C. Left, training and testing timeline. Middle, multisensory protocols: C protocol as described above; Congruent protocol using the same colour (C-sc) combined with different odours as CS+ and CS− during training and testing. Right, the C protocol using distinct colour+odour combinations for CS+ vs CS− resulted in higher immediate memory performance than the C-sc protocol using the same colour with both odours. d. Left, training and testing timeline. Middle, multisensory protocols: Olfactory (O) learning as described above; Multisensory Retrieval (MSR): odours were CS+ and CS− during training and these same odours were combined with different colours during testing. Right, immediate memory performance evoked by MSR was not significantly reduced to that following for Olfactory learning and retrieval. e. Left, training and testing timeline. Middle, multisensory protocols: Congruent (C) protocol as described above; Odour Retrieval (OR): colours+odours were CS+ and CS− during training and only odours were used during testing. Right, flies trained with multisensory stimuli performed better if they were tested with congruent multisensory stimuli compared to only one modality (in this case odour). Asterisks denote significant differences (P < 0.05). Data presented as mean ± standard error of mean (SEM). Individual data points displayed as dots correspond to independent experiments. Groups compared using one-way ANOVA with Tukey’s test (a), unpaired two-sided t-test (b, c, e) and unpaired two-sided Mann-Whitney U-test (d), exact P values and comparisons are given in Supplementary Table 2. N values for each experiment are: a, b, e, n=10; c, n=8; d, n=10 for OR and n=8 for MSR.

Constitutively blocking γd or αβp KC output impairs the visual component of multisensory memories.

a. Top left, schematic of γd KCs. Bottom left, training and testing timeline with constant restrictive temperature (dashed line). Right, 6 h memory performance following Olfactory learning is unchanged when γd KCs are blocked through the experiment using MB607B-GAL4; UAS-Shits1 b and c. Blocking γd KCs throughout the experiment significantly impaired memory in the Congruent protocol (b). The release from the interference effect of the Incongruent protocol did not reach significance (c). d. Top left, schematic of αβp KCs. Bottom left, training and testing timeline with constant restrictive temperature (dashed line). Right, blocking αβp KC output throughout the experiment using c708a-GAL4; UAS-Shits1 did not impair 6 h Olfactory learning. e and f. Memory performance for Congruent (e) and Incongruent (f) protocols changed significantly when αβp KCs were blocked during the experiment. g and h. Top, training and testing timeline with temperature shifting (dashed line). Blocking αβp (g) and γd (h) KCs impaired memory retrieved with Visual cues. h. Top left, schematic of γd KCs. i. Top, training and testing timeline with constant permissive temperature (dashed line), applies to i-l. i-l Memory performance in MB607B-GAL4; UAS-Shits1 flies following Congruent (i), Incongruent (j), Olfactory Retrieval (k) and Visual Retrieval (l) protocols was not affected when training and testing was performed at 23 °C. m. Top left, schematic of αβp KCs. Bottom left, training and testing timeline with constant permissive temperature (dashed line). m-o Memory performance in c708a-GAL4; UAS-Shits1 flies after Congruent (m), Incongruent (n), and Visual Retrieval (o) protocols was not affected when training and testing was at 23 °C. Asterisks denote significant differences (P < 0.05). Data presented as mean ± standard error of mean (SEM). Individual data points displayed as dots correspond to independent experiments. Groups compared using one-way ANOVA with Tukey’s test (a-g, i-o), and Kruskal-Wallis H-test with Dunn’s test (h), exact P values and comparisons are given in Supplementary Table 2. N values for each experiment are: a-c, g, h, n=12; d, n=12 for c708a and n=10 for all other groups, f, n=10; i-o, n=8.

Roles for DPM and APL neurons in multisensory memories.

a and b. Top, Anatomy of APL neuron. Training and testing timelines at constant permissive temperature (dashed line). No defect was observed for Olfactory Retrieval of appetitive multisensory memory (a) or of memory following Olfactory learning (b) in flies with repressed expression of Dop2R RNAi in APL neurons using tubPGAL80ts; VT43924-GAL4.2. c and d. 6h Olfactory Retrieval of appetitive multisensory memory (d) and of memory following Olfactory learning (b) was not affected by adult-restricted RNAi knockdown of DopEcR in APL neurons using tubPGAL80ts; VT43924-GAL4.2. e. Top, Anatomy of DPM neuron. Top, training and testing timeline. Bottom, Multisensory memory performance evoked by Olfactory Retrieval of VT64246-GAL4; UAS-Shits1 flies was unaffected when flies were trained and tested at permissive 23°C. f. Top, training and testing timeline with temperature shifting (dashed line). Bottom, blocking output from DPM neurons during testing impaired the Visual Retrieval of multisensory memory. g. Top, training and testing timeline. Bottom, Multisensory memory performance evoked by Visual Retrieval of VT64246-GAL4; UAS-Shits1 flies was unaffected when flies were trained and tested at permissive 23°C. h. Top, training and testing timeline with temperature shifting (dashed line). Bottom, blocking output from DPM neurons during unisensory training and testing did not affect Olfactory learning performance. i. Top left, anatomy of γ KCs. Top, training and testing timelines. Bottom, Multisensory memory performance evoked by Olfactory Retrieval (i) or Visual Retrieval (j) of MB009B-GAL4; UAS-Shis1 flies was unaffected when trained and tested at 23°C. k. Top left, Anatomy of γd KCs. Bottom left, training and testing timeline. Right, 6h Olfactory learning performance was unaffected in flies with RNAi knockdown of 5-HT2A with MB607B-GAL4. Asterisks denote significant differences (P < 0.05). Data presented as mean ± standard error of mean (SEM). Individual data points displayed as dots correspond to independent experiments. All groups compared using one-way ANOVA with Tukey’s test, exact P values and comparisons are given in Supplementary Table 2. N values for each experiment are: a, b, n=18; c, d, n=10; eg, n=8; h, n=12; i-k, n=8.

Multisensory aversive learning enhances memory for the combined and individual odour and colour cues.

a. Top, aversive training and testing timeline. Bottom, multisensory experimental conditions. Green and blue squares represent colours, light and dark gray represent OCT and MCH odours. Visual (V) learning; Olfactory (O) learning; Congruent (C) protocol; Incongruent (I) protocol; Olfactory Retrieval (OR); Visual Retrieval (VR). b-d. Top, training and testing timelines. Bottom, aversive memory with C protocol was significantly increased to that with I protocol both immediately (b) and 3 h (c) after training. Flies in the C protocol outperformed those tested 3 h after Olfactory (O) Learning. Only the C protocol generated significant 24 h memory performance (d. e and f. Top, training and testing timelines. Bottom, when tested immediately (e) multisensory memory retrieved with colours (VR) was markedly better than that following unisensory Visual learning (V). At 3 h (f) multisensory trained flies performed significantly better when their memory was retrieved with only Olfactory (OR) or Visual (VR) cues than flies trained with unisensory Olfactory (O) or Visual (V) learning. g. Top left, schematic of γd KCs. Bottom left, training and testing timeline with temperature shifting (dashed line). g-i. Blocking output of γd KCs during testing using MB607B-GAL4; UAS-Shits1 alters 3 h memory performance in the Congruent (g), Incongruent (h) and Olfactory Retrieval (I) protocols. j. Top left, schematic of γd KCs. Bottom left, training and testing timeline with constant permissive temperature (dashed line). j-l Memory performance in MB607B-GAL4; UAS-Shits1 flies after Congruent (j), Incongruent (k), and Olfactory Retrieval (l) protocols was not affected when flies were trained and tested at 23°C. Asterisks denote significant differences (P < 0.05). Data presented as mean ± standard error of mean (SEM). Individual data points displayed as dots correspond to independent experiments. m and n. Groups compared using one-way ANOVA with Tukey’s test (b-d, g-l), unpaired two-sided t-test (e, f), exact P values and comparisons are given in Supplementary Table2. N values for each experiment are: b, g, i-l, n=8; c, e, f, h, n=10; d, n=12.