Figures and data in Top-down feedback in deep neural networks leads to functional differences during audiovisual integration

Figures
Tables
Additional files

6 figures, 6 tables and 1 additional file

Figures

Figure 1 with 1 supplement

Download asset Open asset

Description of models.

(a) Each area of the model receives driving feedforward input and modulatory feedback input. Feedback input alters the gain of a neuron, but it does not affect its threshold of activation (multiplicative feedback). In later experiments, we explore an alternative mechanism, where it weakly affects the threshold of activation (composite feedback). (b) Modeled regions and their externopyramidization values (i.e. thickness and relative differentiation of supragranular layers, used as a proxy measure for sensory-associational hierarchical position). Note higher overall externopyramidization values in the occipital lobe compared to the temporal lobe. (c) Using the above hierarchical measures, we constructed models where each connection has a direction (i.e. regions send either feedforward or feedback connections to other regions). In the brainlike model based on human cytoarchitectural data, where all visual regions send feedforward to and receive feedback from the auditory regions. In the reverse model, all auditory regions send feedforward to and receive feedback from visual regions, while connections within a modality remain the same. (d) The resulting artificial neural network (ANN). Outputs of image identification tasks are read out taken from IT, outputs of audio identification tasks are read out from A4, an auditory associational area. Connections between modules are simplified for illustration.

Figure 1—figure supplement 1

Download asset Open asset

Connectivity of the three random models.

The connectivity of the random models was consistent throughout all experiments.

Figure 2

Download asset Open asset

Multimodal visual tasks.

(a) Training conditions. Models must identify the visual stimulus given an ambiguous image and a matching audio clue (VS1) or unambiguous image and distracting audio (VS2). (**b, c**) Accuracy across epochs for tasks VS1 and VS2 on holdout datasets, (d) Trained models were given an ambiguous visual stimulus and a nonmatching audio stimulus (VS3) to assess which modality they align most closely with. (e) Alignment of trained models across epochs based on task VS3. Scenario was shown to model at the end of each training epoch, but never during training. (f) Models were additionally trained and tested on image stimuli only to assess their baseline performance (VS4). (g) Accuracy of models across epochs on task VS4. N=10 for all experiments.

Figure 3 with 2 supplements

Download asset Open asset

Multimodal auditory tasks.

(a) Training conditions. Models must identify the auditory stimulus given an ambiguous audio and a matching visual clue (AS1) or unambiguous audio and distracting image (AS2). (**b, c**) Accuracy across epochs for tasks AS1 and AS2 on holdout datasets, (d) Trained models were given an ambiguous audio stimulus and a nonmatching visual stimulus (AS3) to assess which modality they align most heavily with. (e) Alignment of trained models across epochs based on task AS3. Scenario was shown to model at the end of each training epoch, but never during training. (f) Models were additionally trained and tested on audio stimuli only to assess their baseline performance (AS4), (g) Accuracy of models across epochs on task AS4. N=10 for all experiments.

Figure 3—figure supplement 1

Download asset Open asset

Performance of larger model (hidden state size 32) on AS tasks.

Performance is increased across the board, but difference in stimuli preference is still visible.

Figure 3—figure supplement 2

Download asset Open asset

Effect of process time on AS task performance.

Final epoch accuracy in model given short (3P), standard (7P, equal to number of model layers), and long (10P) processing times when given (a) ambiguous audio, helpful visual clue, (b) clean audio and distracting image, and (c, d) ambiguous audio and nonmatching visual input.

Figure 4

Download asset Open asset

Composite versus multiplicative feedback in multimodal tasks.

(**a, c, e**) Test performances of models with composite feedback and drive-only models trained on visual tasks (VS1 and VS2). The final epoch accuracy of models with composite feedback (C) is compared to that of models with multiplicative feedback (M) shown in previous figures (**b, d, f**) Test performances of models trained on auditory tasks (AS1 and AS2).

Figure 5 with 1 supplement

Download asset Open asset

Audiovisual switching task.

(a) All models were given a new audiovisual output area (AV) connecting to IT and A4. (b) The output area receives an attention flag telling which stream of information to attend to (visual or auditory). The models with feedback use composite feedback. (**c, d**) Test performance of models with composite feedback and drive-only models on all tasks. The models were trained simultaneously on all tasks. (e) Alignment of models given ambiguous visual and ambiguous audio input with differing labels. N=10 for all experiments.

Figure 5—figure supplement 1

Download asset Open asset

Models with multiplicative feedback on the audiovisual switching task.

The task and the drive-only (blue) models are the same between this figure and Figure 5.

Figure 6

Download asset Open asset

Model activity during multimodal tasks.

(a) Information flows through the model from area to area across time. At the first time step, only the primary visual and auditory areas will process information. The areas they feed forward to are activated at the next time step, incorporating top-down information if there is any. (b) Comparison of t-SNE reduced latent space and clustering metric in three areas of the brainlike model at different time stages on task VS2 (ignore audio stimulus). (c) Neighborhood Hit scores in all areas of the trained models across time. Trained models were taken from experiments in Figure 4.

Tables

Table 1

Parameters for the modified ConvGRU equations.

Parameter	Definition
$r_{l}$	Reset gate for layerl
$u_{l}$	Update gate for layerl
$h_{l}$	Hidden state of layerl
$h_{l - 1}$	Feedforward input from the previous layer
$h_{l + 1}$	Feedback input from the next layer
$m$	Modulatory top-down signal
$m_{m o d}$	Modulatory component of top-down signal (composite feedback)
$m_{d}$	Driving component of top-down signal (composite feedback)
$c$	Candidate activation
$W_{r}$	Learnable weight matrix for reset gate
$W_{u}$	Learnable weight matrix for update gate
$W_{c}$	Learnable weight matrix for candidate activation
$W_{m}$	Learnable weight matrix for modulatory signal
$b_{r}$	Bias vector for reset gate
$b_{u}$	Bias vector for update gate
$b_{c}$	Bias vector for candidate activation
$b_{m}$	Bias vector for modulatory signal
$σ$	Sigmoid activation function
$\tanh$	Hyperbolic tangent activation function
ReLU	Rectified Linear Unit activation function
$[x; y]$	Concatenation along the channel dimension
$⋆$	Convolution operation
$⊙$	Element-wise multiplication

Appendix 1—table 1

Mean and standard deviation of final epoch accuracy across VS tasks for small models with multiplicative feedback (Figure 2).

	Brainlike	Reverse	Rand1	Rand2	Rand3
VS1	0.868±0.009	0.914±0.011	0.885±0.013	0.880±0.030	0.836±0.011
VS2	0.933±0.006	0.945±0.013	0.909±0.021	0.884±0.058	0.800±0.052
VS3 (image align)	0.446±0.007	0.440±0.006	0.436±0.010	0.424±0.016	0.406±0.027
VS3 (audio align)	0.216±0.008	0.242±0.011	0.237±0.007	0.247±0.008	0.255±0.029
VS4 (image recognition)	0.964±0.003	0.939±0.007	0.967±0.002	0.907±0.025	0.959±0.009

Appendix 1—table 2

Mean and standard deviation of final epoch accuracy across AS tasks for small models with multiplicative feedback (Figure 3).

	Brainlike	Reverse	Rand1	Rand2	Rand3
AS1	0.972±0.006	0.880±0.046	0.968±0.039	0.879±0.047	0.887±0.054
AS2	0.944±0.016	0.937±0.009	0.945±0.005	0.945±0.015	0.946±0.007
AS3 (image alignment)	0.509±0.066	0.164±0.102	0.452±0.181	0.165±0.095	0.204±0.137
AS3 (audio alignment)	0.326±0.053	0.775±0.155	0.428±0.207	0.764±0.157	0.729±0.187
AS4 (audio recognition)	0.928±0.012	0.938±0.010	0.936±0.016	0.919±0.022	0.922±0.015

Appendix 1—table 3

Mean and standard deviation across VS tasks for small models with composite feedback (Figure 4).

	Brainlike	Reverse	Rand1	Rand2	Rand3
VS1	0.928±0.014	0.907±0.007	0.946±0.007	0.945±0.008	0.941±0.012
VS2	0.949±0.010	0.941±0.012	0.962±0.007	0.671±0.402	0.811±0.317
VS3 (image align)	0.442±0.007	0.445±0.012	0.450±0.006	0.329±0.163	0.384±0.128
VS3 (audio align)	0.247±0.012	0.235±0.011	0.247±0.008	0.483±0.326	0.368±0.260
AS1	0.967±0.007	0.897±0.054	0.964±0.038	0.892±0.053	0.877±0.059
AS2	0.928±0.013	0.928±0.008	0.940±0.011	0.935±0.010	0.933±0.014
AS3 (image align)	0.453±0.073	0.200±0.112	0.444±0.137	0.193±0.128	0.192±0.130
AS3 (audio align)	0.368±0.065	0.722±0.167	0.423±0.164	0.741±0.173	0.741±0.189

Appendix 1—table 4

Mean and standard deviation of final epoch accuracy across audiovisual switching tasks (Figure 5).

	Brainlike	Reverse	Rand1	Rand2	Rand3
VS1	0.909±0.020	0.869±0.023	0.912±0.022	0.883±0.027	0.875±0.028
VS2	0.864±0.042	0.285±0.254	0.874±0.055	0.399±0.306	0.430±0.319
AS1	0.982±0.008	0.900±0.058	0.983±0.014	0.917±0.062	0.913±0.065
AS2	0.897±0.029	0.925±0.023	0.922±0.018	0.937±0.017	0.937±0.019
Image align	0.619±0.044	0.228±0.138	0.570±0.073	0.273±0.155	0.286±0.163
Audio align	0.228±0.050	0.665±0.193	0.273±0.083	0.611±0.206	0.600±0.242

Appendix 1—table 5

Mean and standard deviation of final epoch accuracy for big models on AS tasks (Figure 3—figure supplement 1).

	Brainlike	Reverse	Rand1	Rand2	Rand3
AS1	0.987±0.004	0.949±0.036	0.984±0.008	0.979±0.008	0.977±0.010
AS2	0.952±0.011	0.949±0.009	0.953±0.011	0.962±0.012	0.971±0.002
AS3 (image align)	0.556±0.036	0.201±0.128	0.358±0.102	0.366±0.045	0.353±0.105
AS3 (audio align)	0.280±0.015	0.689±0.203	0.477±0.124	0.431±0.045	0.484±0.135

Additional files

MDAR checklist: https://cdn.elifesciences.org/articles/105953/elife-105953-mdarchecklist1-v1.pdf
Download elife-105953-mdarchecklist1-v1.pdf

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Mendeley

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)

Mashbayar Tugsbayar
Mingze Li
Eilif B Muller
Blake Richards

(2026)

Top-down feedback in deep neural networks leads to functional differences during audiovisual integration

eLife 14:RP105953.

https://doi.org/10.7554/eLife.105953.3

Share this article

Cite this article

Description of models.

Connectivity of the three random models.

Multimodal visual tasks.

Multimodal auditory tasks.

Performance of larger model (hidden state size 32) on AS tasks.

Effect of process time on AS task performance.

Composite versus multiplicative feedback in multimodal tasks.

Audiovisual switching task.

Models with multiplicative feedback on the audiovisual switching task.

Model activity during multimodal tasks.

Parameters for the modified ConvGRU equations.

Mean and standard deviation of final epoch accuracy across VS tasks for small models with multiplicative feedback (Figure 2).

Mean and standard deviation of final epoch accuracy across AS tasks for small models with multiplicative feedback (Figure 3).

Mean and standard deviation across VS tasks for small models with composite feedback (Figure 4).

Mean and standard deviation of final epoch accuracy across audiovisual switching tasks (Figure 5).

Mean and standard deviation of final epoch accuracy for big models on AS tasks (Figure 3—figure supplement 1).

MDAR checklist

Downloads (link to download the article as PDF)

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)