Classic experiments on the temporal determinants of audiovisual integration usually manipulate the lag between the senses and assess the perception of synchrony, temporal order, and audiovisual speech integration (as measured in humans with the McGurk illusion, see Video 1) through psychophysical forced-choice tasks (Venezia et al., 2016, Vroomen and Keetels, 2010; Parise et al., 2025). Among them, we obtained both the audiovisual footage and the psychophysical data from 43 experiments in humans that used ecological audiovisual stimuli (real-life recordings of, e.g. speech and performing musicians, Figure 2A, Figure 2—figure supplement 1 and Supplementary file 1, for the inclusion criteria, see Methods): 27 experiments were simultaneity judgments (van Wassenhove et al., 2007; Lee and Noppeney, 2011; Vroomen and Stekelenburg, 2011; Magnotti et al., 2013; Roseboom and Arnold, 2011; Yuan et al., 2014; Ikeda and Morishita, 2020; van Laarhoven et al., 2019; Lee and Noppeney, 2014) , 10 temporal order judgments (Vroomen and Stekelenburg, 2011, Freeman et al., 2013), six others assessed the McGurk effect (van Wassenhove et al., 2007, Yuan et al., 2014, Freeman et al., 2013).

Figure 2 with 5 supplements see all

Download asset Open asset

For each of the experiments, we can feed the stimuli to the model (Figure 1B and D), and compare the output to the empirical psychometric functions (Equation 10, for details see Methods) (Parise and Ernst, 2016, Pesnot Lerousseau et al., 2022; Parise and Ernst, 2025; Horsfall et al., 2021). Results demonstrate that a population of MCDs can broadly account for audiovisual temporal perception of ecological stimuli, and near-perfectly (rho = 0.97) reproduces the empirical psychometric functions for simultaneity judgments, temporal order judgments, and the McGurk effect (Figure 2A, Figure 2—figure supplement 1). To quantify the impact of the low-level properties of the stimuli on the performance of the model, we ran a permutation test, where psychometric functions were predicted from mismatching stimuli (see Methods). The psychometric curves predicted from the matching stimuli provided a significantly better fit than mismatching stimuli (p<0.001, see Figure 2—figure supplement 1K). This demonstrates that our model captures the subtle effects of how individual features affect observed responses, and it highlights the role of low-level stimulus properties on multisensory perception. All analyses performed so far relied on psychometric functions averaged across observers; individual observer analyses are included in the Figure 2—figure supplements 3–5.

When estimating the perceived timing of audiovisual events, it is important to consider the different propagation speeds of light and sound, which introduce audio lags that are proportional to the observer’s distance from the source (Figure 2B, right). Psychophysical temporal order judgments demonstrate that, to compensate for these lags, humans scale subjective audiovisual synchrony with distance (Figure 2B; Alais and Carlile, 2005). This result has been interpreted as evidence that humans exploit auditory spatial cues, such as the direct-to-reverberant energy ratio (Figure 2B, left), to estimate the distance of the sound source and adjust subjective synchrony by scaling distance estimates by the speed of sound (Alais and Carlile, 2005). When presented with the same stimuli, our model also predicts the observed shifts in subjective simultaneity (Figure 2B, centre). However, rather than relying on explicit spatial representations and physics simulations, these shifts emerge from elementary analyses of natural audiovisual signals. Specifically, in reverberant environments, the intensity of the direct portion of a sound increases with source proximity, while the reverberant component remains constant. As a result, the envelopes of sounds originating close to the observers are more front-heavy than distant sounds (Figure 2B, left). These are low-level acoustic features that the lag detector of the MCD is especially sensitive to, thereby providing a computational shortcut to explicit physics simulations. A Matlab implementation of this simulation is included in Source code 1.

In recent years, audiovisual timing has been systematically studied also in rats (Mafi et al., 2022, Schormans and Allman, 2018, Al Youzbaki et al., 2023, Schormans and Allman, 2023, Schormans et al., 2016, Paulcan et al., 2023), generally using minimalistic stimuli (such as clicks and flashes), and under a variety of manipulations of the stimuli (e.g. loudness) and pharmacological interventions (e.g. GABA and glutamatergic inhibition). Therefore, to further generalize our model to other species, we assessed whether it can also account for rats’ behaviour in synchrony and temporal order judgments. Overall, we could tightly replicate rats’ behaviour (rho = 0.981; see Figure 2C-E, Figure 2—figure supplement 2), including the effect of loudness on observed responses (Figure 2D). Interestingly, the unimodal temporal constants for rats were 4 times faster than for humans: such a different temporal tuning is reflected in higher sensitivity in rats for short lags (<0.1 s), and in humans for longer lags (Figure 2C). This fourfold difference in temporal tuning between rats and humans closely mirrors analogous interspecies differences in physiological rhythms, such as heart rate (~4.7 times faster in rats) and breathing rate (~6.3 times faster in rats) (Agoston, 2017).

While tuning the temporal constants of the model was necessary to account for the difference between humans and rats, this was not necessary to reproduce pharmacologically-induced changes in audiovisual time perception in rats (Figure 2E, Figure 2—figure supplement 2F-G), which could be accounted for solely by changes in the decision-making process(Equation 10). This suggests that the observed effects can be explained without altering low-level temporal processing. However, this does not imply that such changes did not occur—only that they were not required to reproduce the behavioural data in our simulations. Future studies using richer temporal stimuli—such as temporally modulated sequences that vary in frequency, rhythm, or phase—will be necessary to disentangle sensory and decisional contributions, as these stimuli can more selectively engage low-level temporal processing and better reveal whether perceptual changes arise from early encoding or later interpretive stages.

An asset of a low-level approach is that it allows one to inspect, at the level of individual pixels and frames, the features of the stimuli that determine the response of the model (i.e. the saliency maps). This is illustrated in Figure 3 and Videos 1 and 2 for the case of audiovisual speech, where model responses cluster mostly around the mouth area and (to a lesser extent) the eyes. These are the regions where pixels’ luminance changes in synch with the audio track.

Figure 3

Download asset Open asset

Video 1

Download asset

Video 2

Download asset

Classic experiments on the spatial determinants of audiovisual integration usually require observers to localize the stimuli under systematic manipulations of the discrepancy and reliability (i.e. precision) of the spatial cues (Alais and Burr, 2004; Figure 4). This allows one to assess how unimodal cues are weighted and combined to give rise to phenomena such as the ventriloquist illusion (Stratton, 1897). When the spatial discrepancy across the senses is low, observers’ behaviour is well described by Maximum Likelihood Estimation (MLE; Alais and Burr, 2004), where unimodal information is combined in a statistically optimal fashion, so as to maximize the precision (reliability) of the multimodal percept (see Equations 11–14, Methods).

Figure 4 with 1 supplement see all

Download asset Open asset

Given that both the MLE and the MCD operate by multiplying unimodal inputs (see Methods), the time-averaged MCD population response (Equation 16) is equivalent to MLE (Figure 4A). This can be illustrated by simulating the experiment of Alais and Burr, 2004 using both models. In this experiment, observers had to report whether a probe audiovisual stimulus appeared left or right of a standard. To assess the weighing behaviour resulting from multisensory integration, they manipulated the spatial reliability of the visual stimuli and the disparity between the senses (Figure 4B). Figure 4C shows that the integrated percept predicted by the two models is statistically indistinguishable. As such, a population of MCDs (Equation 16) can jointly account for the observed bias and precision of the bimodal percept (Figure 4D–E), with zero parameters. A MATLAB implementation of this simulation is included as Source code 1.

While fusing audiovisual cues is a sensible solution in the presence of minor spatial discrepancies across the senses, integration eventually breaks down with increasing disparity (Chen and Vroomen, 2013)—when the spatial (or temporal) conflict is too large, visual and auditory signals may well be unrelated. To account for the breakdown of multisensory integration in the presence of intersensory conflicts, Körding and colleagues proposed the influential Bayesian Causal Inference (BCI) model (Körding et al., 2007), where uni- and bimodal location estimates are weighted based on the probability that the two modalities share a common cause (Equation 17). The BCI model was originally tested in an experiment in which sound and light were simultaneously presented from one of five random locations, and observers had to report the position of both modalities (Körding et al., 2007; Figure 4G). Results demonstrate that visual and auditory stimuli preferentially bias each other when the discrepancy is low, with the bias progressively declining as the discrepancy increases.

Also a population of MCDs can compute the probability that auditory and visual stimuli share a common cause (Figure 1B and D; Figure 4F, Equation 19), therefore, we can test whether it can also implement BCI. For that, we simulated the experiment of Körding and colleagues, and fed the stimuli to a population of MCDs (Equations 18-20) which near-perfectly replicated the empirical data (rho = 0.99)–even slightly outperforming the BCI model. A MATLAB implementation of this simulation is included as Source code 1.

To test the generalizability of these findings across species and behavioural paradigms, we simulated an experiment in which monkeys (Macaca mulatta) and humans directed their gaze toward audiovisual stimuli presented at varying spatial disparities (Figure 4H; Mohl et al., 2020). If observers infer a common cause, they tend to make a single fixation; otherwise, two—one for each modality. As expected, the probability of a single fixation decreased with increasing disparity (Figure 4H, right). This pattern was captured by a population of MCDs: $MC{D}_{corr}$ values were used to fit the probability of single vs. double saccades as a function of disparity (Equation 21, Figure 4H, right). Critically, using this fit, the model was then able to predict the full distribution of gaze locations (Equation 20, Figure 4H, left) in both species with zero additional free parameters. A Matlab implementation of this simulation is included as Source code 1.

Taken together, these simulations show that behaviour consistent with BCI and MLE naturally emerges from a population of MCDs. Unlike BCI and MLE, however, the MCD population model is both image- and sound-computable, and it explicitly represents the spatiotemporal dynamics of the process (Figure 4A, bottom; Figure 3B; Figure 5; Figure 6B–C). On one hand, this enables the model to be applied to complex, dynamic audiovisual stimuli—such as real-life videos—that were previously off limits to traditional BCI and MLE frameworks, whose probabilistic, non–stimulus-computable formulations prevent them from operating directly on such inputs. On the other, it permits direct, time-resolved comparisons between model responses and neurophysiological measures (Pesnot Lerousseau et al., 2022).

Figure 5

Download asset Open asset

Figure 6

Download asset Open asset

As a practical demonstration, we applied the model (Equation 6) to a real-life video of a performing ventriloquist (Figure 5). The population response dynamically tracked the active talker, clustering around the dummy’s face whenever it produced speech Video 3.

Video 3

Download asset

Multisensory stimuli are typically salient, and a vast body of literature demonstrates that spatial attention is commonly attracted to audiovisual stimuli (Talsma et al., 2010). This aspect of multisensory perception is naturally captured by a population of MCDs, whose dynamic response explicitly represents the regions in space with the highest audiovisual correspondence for each point in time. Therefore, for a population of MCDs to provide a plausible account for audiovisual integration, such dynamic saliency maps should be able to predict human audiovisual gaze behaviour, in a purely bottom-up fashion and with no free parameters. Figure 6A shows the stimuli and eye-tracking data from the experiment of Coutrot and Guyader, 2015, in which observers passively watched a video of four persons talking. Panel B shows the same eye-tracking data plotted over the corresponding MCD population response: across 20 observers, and 15 videos (for a total of over 16,000 frames), gaze was on average directed towards the locations (i.e. pixels) yielding the top 2% MCD response (Figure 6D, Equations 22; 23). The tight correspondence of predicted and empirical salience is illustrated in Figure 6C and Video 4: note how population responses peak based on the active speaker.

Video 4

Download asset

The architecture of each MCD unit used here is the same as described in Parise and Ernst, 2025, however, units here receive time-varying visual and auditory input from spatiotopic receptive fields. The input stimuli ($s$) consist of luminance level and sound amplitude varying over space and time and are denoted as $s_m\left(x,y,t\right)$ – with $x$ and $y$, representing the spatial coordinates along the horizontal and vertical axes, $t$ is the temporal coordinate, and $m$ is the modality (video and audio). When the input stimulus is a movie with mono audio, the visual input to each unit is a signal representing the luminance of a single pixel over time, while the auditory input is the amplitude envelope (later, we will consider more complex scenarios where auditory stimuli are also spatialized).

Each unit operates independently and detects changes in unimodal signals over time by temporal filtering based on two biphasic impulse response functions that are 90° out of phase (i.e. a quadrature pair). A physiologically plausible implementation of this process has been proposed by Adelson and Bergen, 1985 and consists of linear filters of the form:

The phase of the filter is determined by $n$, which based on Emerson et al., 1992 takes the values of 6 for the fast filter and 9 for the slow one. The temporal constant of the filters is determined by the parameter ${\tau }_{bp}$; in humans, its best fitting value is 0.045 s for vision and 0.0367 s for audition. In rats, the fitted temporal constant for vision and audition are nearly identical and their value is 0.010 s.

Fast and slow filters are applied to each unimodal input signal and the two resulting signals are squared and then summed. After that, a compressive non-linearity (square-root) is applied to the output, so as to constrain it within a reasonable range (Adelson and Bergen, 1985). Therefore, the output of each unimodal unit feeding into the correlation detector takes the following form.

where $mod=vid,aud$ represents the sensory modality and * is the convolution operator.

As in the original version (Parise and Ernst, 2016), each MCD consists of two sub-units, in which the unimodal input is low-pass filtered and multiplied as follows.

The impulse response of the low-pass filter of each sub-unit takes the form.

where ${\tau }_{lp}$ represents the temporal constant, and its estimated value is 0.180 s for humans and 0.138 s for rats.

The response of the sub-units is eventually multiplied to obtain $MC{D}_{corr}$, which represents the local spatiotemporal audiovisual correlation, and subtracted to obtain $MC{D}_{lag}$ which describes the relative temporal order of vision and audition

The outputs ${MCD}_{corr}\left(x,y,t\right)$ and ${MCD}_{lag}(x,y,t)$ are the final products of the MCD.

The temporal constants of the filters were fitted using the Bayesian Adaptive Direct Search (BADS) algorithm (Acerbi and Ma, 2017), set to maximize the correlation between the empirical and predicted psychometric functions of the temporal determinants of multisensory integration. For humans, that included all studies in Figure 2—figure supplement 1, besides Patient PH; For rats, it included all non-pharmacological studies in Figure 2—figure supplement 2. To minimize the effect of starting parameters, the fitting was performed 200 times using random starting values (from 0.001 to 1.5 s). The parameters estimated using BADS were further refined using the fminsearch algorithm in MATLAB. Parameter estimation was considerably compute-intensive, hence, the amount of data had to be reduced by rescaling the videos to 15% of the original size (without affecting the frame rate). Besides reducing run-time, this simulates the (Gaussian) spatial pooling occurring in early visual pathways. Details on parameter fitting are provided below.

Now that the MCD population is defined and its parameters are fully constrained, what remains to be explained is how to read out, from the dynamic population responses, the relevant information that is needed to generate a behavioral response, such as eye movements, button presses, nose, or lick contacts, etc. While the MCD units are task-independent and operate in a purely bottom-up fashion, the exact nature of the read-out and decision-making processes depends on the behavioral task, which ultimately determines how to weigh and combine the dynamic population responses. Given that in the present study, we consider different types of behavioral experiments (investigating audiovisual integration through temporal tasks, spatial tasks, and passive observation), the read-out process for each task will be described in separate sections.

The experiments on the temporal constraints of audiovisual integration considered here rely on psychophysical forced-choice tasks to assess the effects of cross-modal lags on perceived synchrony, temporal order, and the McGurk illusion. In humans, such experiments entail pressing one of two buttons; in rats, nose-poking or licking one of two spouts. In the case of simultaneity judgments, on each trial, observers reported whether visual and auditory stimuli appeared synchronous or not $(resp=\left\{yes,no\right\})$. In temporal order judgments, observers reported which modality came first $(resp=\left\{visionfirst,auditionfirst\right\})$. Finally, in the case of the McGurk, observers (humans) had to report what syllable they heard (e.g. ‘da,’ ‘ga,’ or ‘ba,’ usually recoded as ‘fused’ and ‘non-fused percept’ or ‘illusion present’ and ‘illusion absent’). When plotted against lag, the resulting empirical psychometric functions describe how audiovisual timing affects perceived synchrony, temporal order, and the McGurk illusion. Here, we model these tasks using the same read-out and decision-making process.

To account for observed responses, the dynamic, high-bandwidth population responses must be transformed (compressed) into a single number, representing response probability for each lag. In line with standard procedures (Parise and Ernst, 2017), this was achieved by integrating $MC{D}_{corr}\left(x,y,t\right)$ and $MC{D}_{lag}\left(x,y,t\right)$ over time and space, so as to obtain two summary decision variables

and

The temporal window for these analyses consisted of the duration of each stimulus, plus 2 s before and after (during which the audio was silent, and the video displayed a still frame); the exact extension of the temporal window used for the analyses had minimal effect on the results. $\overline{MC{D}_{corr}}$ and $\overline{MC{D}_{lag}}$ are eventually linearly weighted and transformed into response probabilities through a cumulative normal function as follows

Here, $\mathrm{\Phi }$ represents the cumulative normal, ${\beta }_{corr}$ and ${\beta }_{lag}$ are linear coefficients that weigh and scale $\overline{MC{D}_{corr}}$ and $\overline{MC{D}_{lag}}$. ${\beta }_{crit}$ is a bias term, which corresponds to the response criterion. Finally, $p_{MCD}\left(resp\right)$ is the probability of a response, which is $p_{MCD}(Synchronous)$ for the simultaneity judgment task, $p_{MCD}\left(Audiofirst\right)$ for the temporal order judgment task, and $p_{MCD}\left(Fusion\right)$ for the McGurk task. Note how Equation 9 entails that simultaneity judgments, temporal order judgments, and the McGurk illusions are all simulated using the very same model architecture.

With the population model fully constrained (see above), the only free parameters in the present simulations are the ones controlling the decision-making process: ${\beta }_{crit}$, ${\beta }_{corr}$, and ${\beta }_{lag}$. These were separately fitted for each experiment using fitglm MATLAB (binomial distribution and probit link function). For the simulations of the effects of distance on audiovisual temporal order judgments in humans (Alais and Carlile, 2005; Figure 2B), and the manipulation of loudness in rats (Schormans and Allman, 2018; Figure 2D) ${\beta }_{crit}$, ${\beta }_{corr}$, and ${\beta }_{lag}$ were constant across conditions (i.e. distance Alais and Carlile, 2005 or loudness Schormans and Allman, 2018). This way, differences in the psychometric functions as a function of distance (Alais and Carlile, 2005) or loudness (Schormans and Allman, 2018) of the stimuli are fully explained by the MCD. Overall, the model provided a good fit to the empirical psychometric functions, and the average Pearson correlation between human and model response (weighted by sample size) is 0.981 for humans and 0.994 for rats. Naturally, model-data correlation varied across experiments, largely due to sample size. This can be appreciated when the MCD-data correlation for each experiment is plotted against the number of trials for each lag in a funnel plot (see Figure 2—figure supplement 1I). The number of trials for each lag determines the binomial error for data points in Figure 2—figure supplements 1 and 2; accordingly, the funnel plot shows that lower MCD-data correlation is more commonly observed for curves based on smaller sample size. This shows that the upper limit in the MCD-data correlation is mostly constrained by the reliability of the dataset, rather than systematic errors in the model.

To assess the contribution of the low-level properties of the stimuli on model’s performance, we ran a permutation test, where psychometric curves were generated (Equation 10) using stimuli from different experiments (but with the same manipulation of lag). If the low-level properties of the stimuli play a significant role, the correlation between data and model with permuted stimuli should be lower than with non-permuted stimuli. For this permutation, we used the data from simultaneity judgment tasks on humans (Figure 2—figure supplement 1), as with 27 individual experiments, there are enough permutations to render the test meaningful. For that, we used the temporal constants of the MCD fitted before, so that each psychometric curve, each permutation had three free parameters (${\beta }_{crit}$, ${\beta }_{corr}$, and ${\beta }_{lag}$). The results from 200 k permutations demonstrate that the goodness of fit obtained with the original stimuli is superior to that of permuted stimuli. Specifically, the permuted distribution of the mean Pearson correlation of predicted vs. empirical psychometric curves had a mean of 0.972 (σ=0.0036), while such a correlation rose to 0.989 when the MCD received the original stimuli.

Most studies on audiovisual space perception only investigated the horizontal spatial dimension; hence the stimuli can be reduced to $s_m\left(x,t\right)$ instead of $s_m\left(x,y,t\right)$, as in the simulations above (see Equation 2). Additionally, based on preliminary observations, the output ${MCD}_{lag}$ does not seem necessary to account for audiovisual integration in space, hence, only the population response ${MCD}_{corr}(x,t)$ (see Equation 6) will be considered here.

MCD and MLE – simulation of Alais and Burr, 2004. In its general form, the MLE model can be expressed probabilistically as $p_{MLE}\left(x\right)\propto p_{vid}\left(x\right)\bullet p_{aud}\left(x\right)$, where $p_{vid}\left(x\right)$ and $p_{aud}\left(x\right)$ represent the probability distribution of the unimodal location estimate (i.e. likelihood functions), and $p_{MLE}\left(x\right)$ is the bimodal distribution. When the $p_{vid}\left(x\right)$ and $p_{aud}\left(x\right)$ follow a Gaussian distribution, also $p_{MLE}\left(x\right)$ is Gaussian, with the variance (${\sigma }_{MLE}^2$) equal to the product divided by the sum of the unimodal variances:

and the mean consisting of a weighted average of the unimodal means

where

and

To test whether a population of MCDs can replicate the study of Alais and Burr, 2004, this simulation has two main goals. The first one is to compare the predictions of the MCD with that of the MLE model; the second one is to test whether the MCD model can predict observers’ responses.

To simulate the study of Alais and Burr, 2004, we first need to generate the stimuli. The visual stimuli consisted of a 1-D Gaussian luminance profiles, presented for 10ms. Their standard deviations, which determined visual spatial reliability, were defined as the standard deviation of the visual psychometric functions. Likewise, the auditory stimuli also consisted of a 1-D Gaussian sound intensity profile (with a standard deviation determined by the unimodal auditory psychometric function). Note that the spatial reliability in the MCD model jointly depends on the stimulus and the receptive field of the input units; however, teasing apart the differential effects induced by these two sources of spatial uncertainty is beyond the scope of the present study. Hence, for simplicity, here we injected all spatial uncertainty into the stimulus (see also; Parise and Ernst, 2016; Parise and Ernst, 2025). For this simulation, we only used the data from observer LM of Alais and Burr, 2004, as it is the only participant for which the full dataset is publicly available (other observers, however, had similar results).

The stimuli are fed to the model to obtain the population response $MC{D}_{corr}\left(x,t\right)$ (see Equation 6), which is marginalized over time as follows.

This provides a distribution of the population response over the horizontal spatial dimension. Finally, a divisive normalization is performed to transform model response into a probability distribution.

It is important to note that Equations 15; 16 have no free parameters, and all simulations are now performed with a fully constrained model. To test whether the MCD model can perform audiovisual integration according to the MLE model, we replicated the various conditions run by observer LM and calculated the bimodal likelihoods distribution predicted by the MLE and MCD models (i.e. $p_{MLE}\left(x\right)$ and $p_{MCD}\left(x\right)$). These were statistically identical, when considering rounding errors. The results of these simulations are plotted in Figure 4C, and displayed as cumulative distributions as in Figure 1 of Alais and Burr, 2004.

Once demonstrated that $p_{MLE}\left(x\right)=p_{MCD}\left(x\right)$, it is clear that the MCD is equally capable of predicting observed responses. However, for completeness, we compared the prediction of the MCD to the empirical data: the results demonstrate that, just like the MLE model, a population of MCDs can predict both audiovisual bias (Figure 4D) and just noticeable differences (JND, Figure 4E). A MATLAB implementation of this simulation is included as Source code 1.

To account for the spatial breakdown of multisensory integration, the BCI model operates in a hierarchical fashion: first, it estimates the probability that audiovisual stimuli share a common cause $(p\left(C=1\right))$. Next, the model weighs and integrates the unimodal and bimodal information ($p_{mod}\left(x\right)$ and $p_{mle}\left(x\right)$) as follows:

All the terms in Equation 17 have a clear analogue in the computations of the MCD population response: $p_{mle}\left(x\right)$ corresponds to $p_{MCD}\left(x\right)$ (see previous section). Likewise, the homologous of $p_{mod}\left(x\right)$ can be obtained by marginalizing over time and normalizing the output of the unimodal units as follows (see Equation 15)

Finally, the MCD homologous of BCI’s $p(C=1)$ can be read out from the population response following the same logic as Equation 8:

Indeed, the output $\overline{MC{D}_{corr}}$ (Equation 8) not only decreases with increasing temporal disparity (see Figure 1C, left), but it also decreases with increasing spatial disparity (Figure 4F), thereby providing a measure of spatiotemporal coincidence that can be readily transformed into a probability for common cause (Equation 19). Here, we found that including a compressive non-linearity (logarithm of the total ${MCD}_{corr}$ response) provided a tighter fit to the empirical data. With $p_{MCD}\left(C=1\right)$ representing the probability that vision and audition share a common cause, and $p_{MCD}\left(x\right)$ and $\overline{MC{D}_{mod}}\left(x\right)$ representing the bimodal and unimodal population responses, the MCD model can simulate observed responses as follows:

The similarity between Equation 20 and Equation 17 demonstrates the fundamental homology of the BCI and MCD models: what remains to be tested is whether the MCD can also account for the results of Körding et al., 2007. For that, just like the BCI model, the MCD model also relies on four free parameters: two are shared by both models, and represent the spatial uncertainty (i.e. the variance) of the unimodal input (i.e. ${\sigma }_{vid}^2$ and ${\sigma }_{aud}^2$). Additionally, the MCD model then needs two linear coefficients (slope ${\beta }_{corr}$ and intercept ${\beta }_{crit}$) to transform the dynamic population response into a probability of a common cause. Conversely, the remaining two parameters of the BCI model correspond to a prior for common cause and another for central location, neither of which are necessary to account for observed responses in the present framework. Parameters were fitted using BADS (Acerbi and Ma, 2017) set to maximize the Pearson correlation between model and human responses (Figure 4G). Overall, the MCD provided an excellent fit to the empirical data (r=0.99, Figure 4G), even slightly exceeding the performance of the BCI model while relying on the same degrees of freedom. Given that the fitted value of the slope parameter (${\beta }_{corr}$) approached 1, we repeated the fitting while removing ${\beta }_{corr}$ from Equation 19: even with just three free parameters (one fewer than the BCI model), the MCD is in line with the BCI model and the correlation with the empirical data was 0.98. A Matlab implementation of this simulation is included as Source code 1.

Mohl and colleagues used eye movements to test whether humans and monkeys integrate audiovisual spatial cues according to BCI. The targets consisted of either unimodal or bimodal stimuli. Following the same logic as the simulation of Alais and Burr, 2004 (see above), the unimodal input consisted of impulses with a Gaussian spatial profile (Figure 4A–B), whose the variance (i.e., ${\sigma }_{vid}^2$ and ${\sigma }_{aud}^2$) was set equal to the variance of the fixations measured in the unimodal trials (averaged across observers). Although the probability of a single fixation decreases with increasing disparity (Figure 4H, right), observers sometimes failed to make a second fixation even when the stimuli were noticeably far apart (lapses). This was especially true for monkeys, which had a lapse rate of 16%, indicative of low attention and compliance. To account for this, we can modify Equation 19 and obtain the probability of a single fixation as follows:

Here, $p_{lapse}$ is a free parameter that represents the probability of making the incorrect number of fixations, irrespective of the discrepancy. As in Equation 19, ${\beta }_{corr}$ and ${\beta }_{crit}$ are free parameters that transform the dynamic population response into a probability of a common cause (i.e. single fixation). Equation 21 could tightly reproduce the observed probability of a single fixation (Figure 4H, right), and the Pearson correlation between the model and data was 0.995 for monkeys and 0.988 for humans.

With the parameters $p_{lapse}$, ${\beta }_{corr}$ and ${\beta }_{crit}$ fitted to the probability of a single fixation (i.e. the probability of a common cause), it is now possible to predict gaze direction (i.e. the perceived location of the stimuli) with zero free parameters. For that, we can use Equation 21 to get the probability of a common cause and predict gaze direction using Equation 20. The distribution of fixations predicted by the MCD closely follows the empirical histogram in both species (Figure 4H) and the correlation between the model and data was 0.9 for monkeys and 0.93 for humans. Note that Figure 4H shows only a subset of the 20 conditions tested in the experiment (the same subset of conditions shown in the original paper). A MATLAB implementation of this simulation is included as Source code 1.

To test whether the dynamic MCD population response can account for gaze behavior during passive observation of audiovisual footage, a simple solution is to measure whether observers preferentially looked where the ${MCD}_{corr}$ response is maximal. Such an analysis was separately performed for each frame. For that, gaze directions were first low pass filtered with a Gaussian kernel ($\sigma =14pixels$) and normalized to probabilities $p_{gaze}(x,y)$. Next, we calculated the average MCD responses at gaze direction for each frame (t); this was done by weighing ${MCD}_{corr}$ by $p_{gaze}$ as follows

To assess the MCD response at gaze $(\overline{MC{D}_{gaze}}\left(t\right))$ is larger than the frame average, we calculated the standardized mean difference (SMD) for each frame as follows

Across the over 16,000 frames of the available dataset, the average SMD was 2.03 (Figure 6D). Given that the standardized mean difference serves as a metric for effect size, and that effect sizes surpassing 1.2 are deemed very large (Sawilowsky, 2009), it is remarkable that the MCD population model can so tightly account for human gaze behavior in a purely bottom-up fashion and without free parameters.

Much like Bayesian ideal observer models, the present framework can be naturally extended to trimodal integration. As described in Equation 6, the, ${MCD}_{corr}\left(x,y,t\right)$ response is based on the pointwise product of input transients across modalities. In the bimodal case, this corresponds to the product of auditory and visual transient channels. For three modalities (e.g. auditory, visual, tactile), this generalizes to a trimodal coincidence detector, in which MCD units compute:

This detector responds maximally when transients in all three modalities co-occur in time and space. As in the bimodal case (Figure 4), the trimodal MCD response closely approximates the predictions of the maximum likelihood estimation (MLE) model (Figure 4—figure supplement 1).

$MC{D}_{lag}\left(x,y,t\right)$ (Equation 7) is instead defined via opponency (subtraction) between the two subunits of the MCD, which introduces a directional asymmetry between modalities. This structure makes it fundamentally pairwise. As a result, extending $MC{D}_{lag}\left(x,y,t\right)$ to three modalities would require computing pairwise lag estimates (AV, VT, AT) independently.

A variety MATLAB script running the MCD population model is included as Source code 1. This includes a code running the MCD on real-life footage (i.e., this is what was used to model MCD responses to the temporal determinants of multisensory integration, spatial orienting, and Figure 5). Additional codes simulate the experiment of Alais and Carlile, 2005 and all simulations in Figure 5.

The diversity of the stimuli used in this dataset requires some preprocessing before the stimuli can be fed to the population model. First, all movies were converted to grayscale (scaled between 0 and 1) and the soundtrack was converted to rms envelope (scaled between 0 and 1), thereby removing chromatic and tonal information. Movies were then padded with 2 s of frozen frames at onset and offset to accommodate for the manipulation of the lag. Finally, the luminance of the first still frame was set as baseline and subtracted from all subsequent frames (see Matlab codes in the Source code 1). Along with the padding, this helps minimizing transient artefacts induced by the onset of the video.

Video frames were scaled to 15% of the original size, and the static background was cropped. On a practical side, this made the simulations much faster (which is crucial for parameter estimation); on a theoretical side, such a down sampling simulates the Gaussian spatial pooling of luminance across the visual field (unfortunately, the present datasets do not provide sufficient information to convert pixels into visual angles). In a similar fashion, we down sampled sound envelope to match the frame rate of the video.

For the simulations of Horsfall et al., 2021, the envelope of the auditory stimuli was extracted from the waveforms shown in the figures of the original publication. That was done using WebPlotDigitizer to trace the profile of the waveforms; the digitized points were then interpolated and resampled at 1000 Hz. To preserve the manipulation of the direct-to-reverberant waves, the section of the envelope with the reverberant signal was identical across the four conditions (i.e. distances), so that what varied across conditions was the initial portion of the signals (the direct waves). For the simulations of Horsfall et al., 2021, all four psychometric functions were fitted simultaneously, so that the four psychometric functions all relied on just three free parameters: the ones related to the decision-making process. A MATLAB code running this simulation is now included as Source code 1.

Most of the simulations described so far rely on group-level data, where psychometric curves represent the average response across the pool of observers that took part in each experiment. Individual psychometric functions, however, sometimes vary dramatically across observers; hence, one might wonder whether the MCD, besides predicting stimulus-driven variations in the psychometric functions, can also capture individual differences. A recent study by Yarrow and colleagues (Yarrow et al., 2023) directly addressed this question, and concluded that models of the Independent Channels family outperform the MCD at fitting responses individual differences.

Although it can be easily shown that such a conclusion was supported by an incomplete implementation of the MCD (which did not include the ${MCD}_{lag}$ output), a closer look at the two models against the same datasets help us illustrate their fundamental difference and highlight a key drawback of perceptual models that take parameters as input. Therefore, we first simulated the impulse stimuli used by Yarrow and colleagues (Yarrow et al., 2023), fed them to the MCD, and used Equation 10 to generate the individual psychometric curves. Given that their stimuli consisted of temporal impulses with no spatiotemporal manipulation, a single MCD unit is sufficient to run these simulations. Overall, the model provided an excellent fit to the original data and tightly captured individual differences: the average Pearson correlation between predicted and empirical psychometric functions across the 57 curves shown in Figure 2—figure supplement 3 is 0.98. Importantly, for such simulations, the MCD was fully constrained, and the only free parameters (3 in total) were the linear coefficients of Equation 10, which describe how the output of the MCD is used for perceptual decision-making. For comparison, also Independent Channels models achieved analogous goodness of fit, but they required at least five free parameters (depending on the exact implementation Yarrow et al., 2023).

To assess the generalizability of this finding, we additionally simulated the individual psychometric functions from the experiments that informed the architecture of the MCD units used here. Specifically, Parise and Ernst, 2025 run two psychophysical studies using minimalistic stimuli that only varied over time. In the first one, auditory and visual stimuli consisted of step increments and/or decrements in intensity. Audiovisual lag was parametrically manipulated using the method of constant stimuli, and observers were required to perform both simultaneity and temporal order judgments. Following the logic described above, we fed the stimuli to the model, and used Equation 10 (with three free parameters) to simulate human responses. Results demonstrate that the MCD can account for individual differences regardless of the task (Figure 2—figure supplement 4): the average Pearson correlation between empirical and predicted psychometric curves was 0.97 for the simultaneity judgments, and 0.96 for the temporal order judgments (64 individual psychometric curves from eight observers, for a total of 9600 trials). This generalizes the results of the previous simulation to a different type of stimuli (steps vs. impulses) and extends them to include a task, the temporal order judgment, which was not considered by Yarrow and colleagues (Yarrow et al., 2023) (whose model can only perform simultaneity judgments).

The second study of Parise and Ernst, 2025 consists of simultaneity judgments for periodic audiovisual stimuli defined by a square-wave intensity envelope (Figure 2—figure supplement 5). Simultaneity judgments for this type of periodic stimuli are also periodic, with two complete oscillations in perceived simultaneity for each cycle of phase shifts between the senses (a phenomenon known as frequency doubling). Once again, using Equation 10, the MCD could account for individual differences in observed behavior (five psychometric curves from five observers, for a total of 3000 trials) with an average Pearson correlation of 0.93, while relying on just three free parameters.

It is important to note that for these simulations, the same MCD model accurately predicted (in a purely bottom-up fashion) bell-shaped SJ curves for non-periodic stimuli, and sinusoidal curves for periodic stimuli. Alternative models of audiovisual simultaneity that directly take lag as input always enforce bell-shaped psychometric functions, where perceived synchrony monotonically decrease as we move away from the point of subjective simultaneity. As a result, in the absence of ad-hoc adjustments they all necessarily fail at replicating the results of Experiment 2 of Parise and Ernst, 2025, due to their inability to generate periodic psychometric functions. Conversely, the MCD is agnostic regarding the shape of the psychometric functions, hence the very same model used to predict the standard bell-shaped simultaneity judgments of Yarrow et al., 2023 can also predict the periodic psychometric functions of Parise and Ernst, 2025, including individual differences across observers (all while relying on just three free parameters).

To thoroughly compare observed and model behavior, this study requires a large and diverse dataset consisting of both the raw stimuli and observers’ responses. For that, we adopted a convenience sampling and simulated the studies for which both stimuli and responses were available (either in public repositories, shared by the authors, or extracted from published figures). The inclusion criteria depend on what aspect of multisensory integration is being investigated, and they are described below.

For the temporal determinants of multisensory integration in humans, we only included studies that: (1) used real-life audiovisual footage, (2) performed a parametric manipulation of lag, and (3) engaged observers in a forced-choice behavioral task. Forty-three individual experiments met the inclusion criteria (Figure 2A-B, Figure 2—figure supplement 1). These varied in terms of stimuli, observers, and tasks. In terms of stimuli, the dataset consists of responses from 105 unique real-life videos (see Figure 2—figure supplement 1 and Supplementary file 1). The majority of the videos represented audiovisual speech (possibly the most common stimulus in audiovisual research), but they varied in terms of content (i.e. syllables, words, full sentences, etc.), intelligibility (i.e. sine-wave speech, amplitude-modulated noise, blurred visuals, etc.), composition (i.e. full face, mouth-only, oval frame, etc.), speaker identity, etc. The remaining non-speech stimuli consist of footage of actors playing a piano or a flute. The study from Alais and Carlile, 2005 was included in the dataset because, even if the visual stimuli were minimalistic (blobs), the auditory stimuli consisted of ecological auditory depth cues depth cues recorded in a real reverberant environment (the Sydney Opera House, see below for details on the dataset). The dataset contains forced-choice responses to three different tasks: speech categorization (i.e., for the McGurk illusion), simultaneity judgments, and temporal order judgment. In terms of observers, besides the general population, the dataset consists of experimental groups varying in terms of age, musical expertise, and even includes a patient, PH, who reports hearing speech before seeing mouth movements after a lesion in the pons and basal ganglia (Freeman et al., 2013). Taken together, the dataset consists of ~1 k individual psychometric functions, from 454 unique observers, for a total of ~300 k trials; the psychometric curves for each experiment are shown in Figure 2A-B, Figure 2—figure supplement 1. All these simulations are based on psychometric functions averaged across observers, for simulations of individual observers, see Figure 2—figure supplements 3–5.

For the temporal determinants of multisensory integration in rats, we included studies that performed a parametric manipulation of lag and engaged rats in simultaneity and temporal order judgment tasks. Sixteen individual experiments (Mafi et al., 2022, Schormans and Allman, 2018, Al Youzbaki et al., 2023, Schormans and Allman, 2023, Schormans et al., 2016, Paulcan et al., 2023) met the inclusion criteria (Figure 2C-D, Figure 2—figure supplement 2 and): all of them used minimalistic audiovisual stimuli (clicks and flashes) and with a parametric manipulation of audiovisual lag. Overall, the dataset consists of ~190 individual psychometric functions, from 110 rats (and 10 humans), for a total of ~300 k trials.

For the case of the spatial determinants of multisensory integration, to the best of our knowledge, there are no available datasets with both stimuli and psychophysical responses. Fortunately, however, the spatial aspects of multisensory integration are often studied with minimalistic audiovisual stimuli (e.g. clicks and blobs), which can be simulated exactly. Audiovisual integration in space is commonly framed in terms of optimal statistical estimation, where the bimodal percept is modelled either through MLE or BCI. To provide a plausible account for audiovisual integration in space, a population of MCDs should also behave as a Bayesian-optimal estimator. This hypothesis was tested by comparing the population response to human data in the studies that originally tested the MLE and BCI models; hence, we simulated the study of Alais and Burr, 2004 and Körding et al., 2007. Such simulations allow us to compare the data with our model, and our model with previous ones (MLE and BCI). Given that in these two simulations, a population of MCDs behaves just like the MLE and BCI models (and with the same number of free parameters or fewer), the current approach can be easily extended to other instances of sensory cue integration previously modelled in terms of optimal statistical estimation. This was tested by simulating the study of Mohl et al., 2020, who used eye movements to assess whether BCI can account for audiovisual integration in monkeys and humans (Figure 4).

Finally, we tested whether a population of MCDs can predict audiovisual orienting and gaze behaviour during passive observation of ecological audiovisual stimuli. Coutrot and Guyader, 2015 run the ideal testbed for this hypothesis: much like our previous simulations (Figure 2A), they employed audiovisual speech stimuli, recorded indoor, with no camera shake. Specifically, they tracked eye movements from 20 observers who passively watched 15 videos of a lab meeting (see Figure 6A). Without fitting parameters, the population response tightly matched the empirical saliency maps (see Figure 6B–D and Video 4).

The temporal filters determine the temporal tuning of the model and consist of three parameters: two temporal constants of the unimodal band-pass filters (${\tau }_{bpA},{\tau }_{bpV}$; Equation 1), and one bimodal constant (${\tau }_{lp},$ Equation 5). For humans, these parameter values were estimated by combining data from all 40 experiments shown in Figure 2—figure supplement 1, excluding the data from Patient PH. These experiments encompass SJ, TOJ, and McGurk tasks, each involving parametric manipulations of audiovisual lag, as well as tasks using ecological audiovisual stimuli (i.e. real-life footage). For rats, we followed the same approach, combining all psychometric curves shown in Figure 2—figure supplement 2 (excluding pharmacological manipulation experiments), based on stimuli consisting of clicks and flashes in TOJ and SJ tasks.

To simulate each trial, we fed the stimuli into the population model to estimate internal response variables $\overline{MC{D}_{corr}}$ and $\overline{MC{D}_{lag}}$ (Equations 8; 9). Response probabilities were then derived from these internal signals as described in Equation 10 (see also Parise and Ernst, 2025). This decision stage introduces three additional parameters: ${\beta }_{crit}$ (the criterion, a bias term), and ${\beta }_{corr}$, and ${\beta }_{lag}$ (gain parameters, acting as scaling factors). These decision-related parameters were estimated independently for each experiment. An exception was made for simulations based on Schormans and Allman, 2018; Figure 2—figure supplement 2E, which manipulated audio intensity across three levels—here, we used a single set of ${\beta }_{crit}$, ${\beta }_{corr}$, and ${\beta }_{lag}$ values across all conditions.

Two search algorithms were used for parameter estimation. First, we applied a global optimization method—Bayesian Adaptive Direct Search (BADS; Acerbi). To minimize the influence of initial values, we ran the BADS 200 times with different starting points (from 0.001 to 1.5 s). The best-fitting parameter values were then further refined using a local optimizer (fminsearch in MATLAB). The cost function minimized by both algorithms was 1 minus the Pearson correlation between real and simulated data, averaged across all experiments (each experiment was weighted equally, regardless of trial count or sample size). To assess robustness, we repeated the procedure using a mean squared error (MSE) cost function, which produced a set of parameter values (${\tau }_{bpA}$=0.042 s ${\tau }_{bpV}$=0.039 s; ${\tau }_{lp}$=0.156 s) that are closely aligned with the original estimate (${\tau }_{bpA}$=0.045 s ${\tau }_{bpV}$=0.036 s; ${\tau }_{lp}$=0.180 s).

This approach to parameter estimation, which aggregates data from a broad range of paradigms (n=40 for humans, n=15 for rats), enables us to estimate the temporal constants of the MCD units using a variety of ecological audiovisual stimuli (n=105 unique stimuli). Given that the stimuli consist of raw audiovisual footage, parameter estimation is computationally demanding — each search iteration can take up to 10 min. This makes standard techniques such as leave-one-out cross-validation impractical. The generalizability of the estimated temporal constants can nevertheless be tested against new, unseen data.

First, we tested whether the MCD model, with fixed temporal constants, could predict a well-known finding: that perceived audiovisual synchrony varies with distance in reverberant environments (Alais and Carlile, 2005). To do this, we used the previously estimated temporal constants and fitted only a single set of three decision parameters to predict the four psychometric curves in Figure 2B—effectively using fewer than one free parameter per curve. Importantly, no changes were made to the model’s temporal tuning. The MCD was able to fully reproduce the separation of the four curves across conditions. This result is not simply an extension of the model to a new stimulus set. Rather, it provides a purely bottom-up account of how perceived synchrony scales with spatial depth—without invoking any explicit computation of distance or higher-level inference mechanisms.

Next, we examined whether the estimated temporal constants—which had so far only been tested on group-averaged data—could also predict individual-level responses. To do this, we simulated individual observer data using previously published datasets: SJ data from Yarrow et al., 2023, and both TOJ and SJ data from Parise and Ernst, 2025, Experiments 1 and 2. These simulations allowed us to assess whether the model could generalize from group-level patterns to individual-level behaviour. The results confirmed that the MCD, using a single set of fixed temporal constants across all individuals, could successfully predict both individual and group data. Only the decision parameters (${\beta }_{crit}$, ${\beta }_{corr}$, and ${\beta }_{lag}$) were fitted per observer, showing that temporal tuning itself can be considered constant across different observers.

Finally, using the same set of temporal constants, we evaluated whether the $\overline{MC{D}_{corr}}$ population output (Equation 6) could predict gaze behaviour during passive observation of real-life audiovisual footage—this time with zero free parameters. For this, we presented the audiovisual frames to the model (Figure 6A) and computed the MCD response for each pixel and time frame (Figure 6B). Across frames, the model’s output consistently predicted the location of gaze direction: the region of the screen with the highest $\overline{MC{D}_{corr}}$ response systematically attracted participants’ gaze (Figure 6D).

In summary, we have used data from a large pool of experiments consisting of 105 unique stimuli to estimate the temporal constants of the MCD model. These constants, once estimated, demonstrated predictive validity across a wide range of behavioural measures (SJ, TOJ, eye-tracking), stimulus types, species (humans and rats), and data granularities (group and individual). They generalized successfully to completely novel behavioural datasets and natural viewing conditions—including eye-tracking during passive video observation—all without any re-tuning of the model’s core temporal filters.

No external funding was received for this work.

We thank Dr. Irene Senna for insightful comments and continuous support throughout all stages of this study. We are also grateful to Dr. Alessandro Moscatelli for valuable feedback on the manuscript, and to Prof. Marc Ernst for stimulating discussions during the early phases of this work.

1. Sent for peer review: January 31, 2025
2. Preprint posted: February 4, 2025
3. Reviewed Preprint version 1: May 13, 2025
4. Reviewed Preprint version 2: September 3, 2025
5. Version of Record published: November 4, 2025

You can cite all versions using the DOI https://doi.org/10.7554/eLife.106122. This DOI represents all versions, and will always resolve to the latest one.

© 2025, Parise

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.