Our different sensory systems each continuously adapt to changes in the environment (Webster, 2012). Thus, to maintain stable and coherent perception in a multisensory and ever-changing world, the brain needs to dynamically adjust for sensory discrepancies between the different modalities. This process of multisensory recalibration takes place continually and is perhaps more fundamental than multisensory integration because integration would not be beneficial when the underlying cues are biased. While the neural bases of multisensory integration have received a lot of attention (Chen et al., 2013a; Gu et al., 2008; Stein et al., 2014), the neural bases of multisensory recalibration have been explored to a much lesser degree.

Cross-modal recalibration has been observed in a variety of multisensory settings. One well-known example is the ventriloquist aftereffect, in which exposure to a consistent spatial discrepancy between auditory and visual stimuli induces a subsequent shift in the perceived location of sounds (Bertelson and De Gelder, 2004; Canon, 1970; Kramer et al., 2019; Radeau and Bertelson, 1974; Recanzone, 1998; Watson et al., 2021). Also, the rubber-hand illusion leads to an offset in hand proprioception in the direction of the visually observed rubber hand (Abdulkarim et al., 2021; Botvinick and Cohen, 1998; Kennett et al., 2001; Thériault et al., 2022; Tsakiris and Haggard, 2005). Although it was initially thought that only the non-visual cues recalibrate to vision, termed visual dominance (Brainard and Knudsen, 1993; Rock and Victor, 1964), further work in a variety of paradigms has revealed both visual and non-visual recalibration (Atkins et al., 2003; Lewald, 2002; van Beers et al., 2002; Zaidel et al., 2011).

Most of what we know about multisensory recalibration is described at the behavioral level (Lewald, 2002), with little known about its neuronal underpinnings. Recent studies in humans have shed some light on this question. In the ventriloquism aftereffect, cross-modal (audio-visual) recalibration of auditory signals (fMRI) is seen in low-level auditory cortical areas (Zierul et al., 2017). According to that study and another recent (EEG) study (Park and Kayser, 2021), higher-level parietal regions also play a central role in cross-modal spatial recalibration. Moreover, Park and Kayser, 2021 further suggest that frontal regions consolidate the behavioral shift under sustained multisensory discrepancies. However, these methods (fMRI and EEG) lack the resolution to probe recalibration at the level of single neurons.

In a series of classic studies, Kundsen and Brainard investigated multisensory plasticity at the neuronal and circuit levels in the barn owl (Knudsen, 2002; Knudsen and Brainard, 1991; Linkenhoker and Knudsen, 2002). They found profound neuronal plasticity in juvenile owls reared with prismatic lenses that systematically displaced their field of view. In that case, the auditory space map in the optic tectum was recalibrated to be aligned with the displaced visual field (Knudsen and Brainard, 1991). However, multisensory plasticity is not limited to the development, and the neuronal bases of how multiple sensory systems continuously adapt to one another in the adult brain remain fundamentally unknown.

Self-motion perception (the subjective feeling of moving through space) relies primarily on visual and vestibular cues (Butler et al., 2015; Butler et al., 2010; de Winkel et al., 2010; Fetsch et al., 2011; Fetsch et al., 2009; Gu et al., 2007; Warren et al., 1988). Multisensory integration of visual and vestibular signals can improve heading perception (Butler et al., 2015; Dokka et al., 2015; Gu et al., 2008). However, conflicting or inconsistent visual and vestibular information often leads to motion sickness (Oman, 1990; Reason and Brand, 1975). Interestingly, this subsides after prolonged exposure to the sensory motion conflict, presumably through brain mechanisms of multisensory recalibration (Held, 1961; Shupak and Gordon, 2006). Thus, self-motion perception – a vital skill for everyday function with intrinsic plasticity – offers a prime substrate to study cross-modal recalibration.

We previously investigated and found robust perceptual recalibration of both visual and vestibular cues in response to a systematic vestibular–visual heading discrepancy (Zaidel et al., 2011). Similar results were seen for both humans and monkeys. In that paradigm, no external feedback was given. Thus, the need for recalibration arose solely because of the cue discrepancy (we therefore call this condition unsupervised). This led to shifts in subsequent visual and vestibular perceptual estimates toward each other, presumably to reduce the conflict. This is in line with the notion that unsupervised recalibration aims to maintain ‘internal consistency’ between the cues (Burge et al., 2010). However, the neuronal basis of this everyday multisensory plasticity is unknown. This study aimed to test unsupervised recalibration of visual and vestibular neuronal tuning, and how it may differ across multisensory cortical areas.

In line with human neuroimaging studies that showed cross-modal (auditory–visual) recalibration in relatively early sensory areas (Amedi et al., 2002; Zierul et al., 2017), and because unsupervised recalibration is sensory driven (occurs as a result of the cross-modal discrepancy, in the absence of overt feedback) we expected to observe neural correlates of unsupervised visual–vestibular recalibration in relatively early cortical areas with self-motion signals. Previous studies with monkeys identified two relatively early multisensory cortical areas involved in self-motion perception: the medial superior temporal area (Gu et al., 2006) and the parietal insular vestibular cortex (PIVC, Chen et al., 2010). Neurons in MSTd respond to large optic flow stimuli, conducive to the visual perception of self-motion (Gu et al., 2006). Vestibular responses are also present in MSTd, however visual self-motion signals dominate (Gu, 2018; Gu et al., 2012). PIVC has strong vestibular responses, without strong tuning to visual optic flow (Chen et al., 2010).

The ventral intraparietal (VIP) area also has robust responses to visual and vestibular self-motion stimuli, however, it is marked by strong choice signals (Chen et al., 2016; Gu, 2018; Zaidel et al., 2017). It is thus considered a higher-level multisensory area, possibly involved in perceptual decision-making or higher-order perceptual functions. Accordingly, and in line with findings of parietal involvement in human cross-modal recalibration (Park and Kayser, 2021; Zaidel et al., 2021; Zierul et al., 2017), we were interested to see what correlates of unsupervised recalibration we would see in parietal neurons. Different types of multisensory recalibration observed in VIP vs. lower-level (MSTd and PIVC) multisensory areas can provide important insights into their differential underlying functions. Thus, in this study, we focused on these three multisensory cortical areas. We examined whether and how their visual and vestibular neural tuning changed in accordance with corresponding perceptual shifts during a single session (~1 hr) of unsupervised cross-modal recalibration.

Three monkeys performed a heading discrimination task before, during, and after undergoing cross-modal recalibration to spatially conflicting vestibular–visual signals. Simultaneous to behavioral performance, we recorded from single neurons extracellularly in areas MSTd (upper bank of the superior temporal sulcus, N = 83 total; 19 from monkey D, 64 from monkey K), PIVC (upper bank and the tip of the lateral sulcus, N = 160 total; 91 from monkey D, 69 from monkey B), and VIP (lower bank and tip of the intraparietal sulcus, N = 118 total; 103 from monkey D, 15 from monkey B).

The experiment paradigm followed similar methodology as our previous behavioral study (Zaidel et al., 2011). It consisted of three consecutive blocks: pre-recalibration (Figure 1A), recalibration (Figure 1B), and post-recalibration (Figure 1C). In the recalibration block, the monkeys were presented with combined stimuli (simultaneous visual and vestibular cues) with a systematic discrepancy between the visual and vestibular heading directions. In the pre- and post-recalibration blocks, the unisensory perception was measured using visual-only or vestibular-only cues. The effects of recalibration on visual and vestibular perception were measured by the shifts in the post- vs. pre-recalibration psychometric curves. We first (in the next section) present the monkeys’ perceptual recalibration results. Thereafter, we present the neural correlates thereof.

Figure 1

Download asset Open asset

Vestibular and visual perceptual estimates shift toward each other

Figure 2 shows example psychophysical data from two experimental sessions. Replicating our previous behavioral results (Zaidel et al., 2011), we found that both visual and vestibular psychometric functions shifted in the direction required to reduce cue conflict. Namely, when the vestibular and visual heading stimuli were systematically offset, such that they consistently deviated to the right and the left, respectively (Δ⁺, Figure 2A), the vestibular post-recalibration curve (blue) was shifted rightward vs. pre-recalibration (black). Note that a rightward shift of the psychometric curve indicates a leftward perceptual shift (identified by a lower propensity for ‘rightward’ choices at 0° heading for the blue curve). Complementarily, the visual post-recalibration psychometric curve (red) shifted leftward vs. pre-recalibration (black), albeit to a lesser degree, indicating a rightward perceptual shift. In a reverse manner, when the vestibular and visual heading stimuli were offset to the left and right, respectively (Δ⁻, Figure 2B), the vestibular post-recalibration curve (blue) shifted to the left, and the visual post-recalibration curve shifted to the right.

Figure 2

Download asset Open asset

Figure 2—source data 1

Individual monkey summary statistics of behavioral shifts.

https://cdn.elifesciences.org/articles/82895/elife-82895-fig2-data1-v1.doc

Download elife-82895-fig2-data1-v1.doc

These perceptual shifts were quantified by the difference between the post- vs. pre-recalibration curves’ PSEs (points of subjective equality). A psychometric curve’s PSE represents the heading angle of equal right/left choice proportion, that is, the heading that participants would supposedly perceive as straight ahead. The vestibular and visual psychometric shifts were positive and negative, 3.40° and −1.01°, respectively, in Figure 2A, and negative and positive, −3.68° and 1.00°, respectively, in Figure 2B. Thus, in both cases (Figure 2A, B), both the vestibular and the visual cues shifted in the direction required to reduce the cue conflict (i.e., in opposite directions).

In each session, only one discrepancy orientation was tested. Namely, vestibular and visual headings were either offset to the right and left (respectively), or vice versa. These discrepancies were arbitrarily defined as positive (Δ⁺) or negative (Δ⁻), respectively. In total, we collected data from 241 sessions with Δ⁺ and 227 sessions with Δ⁻. Distributions of the vestibular and visual PSE shifts across sessions are presented in Figure 2C (above and below the abscissa for Δ⁺ and Δ⁻, respectively). The vestibular PSEs were shifted significantly to the right for the Δ⁺ condition (mean ± SE = 1.12° ± 0.12°; p = 2.3 × 10⁻¹⁷, paired t-test), and significantly to the left for the Δ⁻ condition (mean ± SE = −1.76° ± 0.14°; p = 1.0 × 10⁻²⁸, paired t-test). The visual PSEs were shifted significantly to the left for the Δ⁺ condition (mean ± SE = −0.73° ± 0.11°; p = 6.5 × 10⁻¹¹, paired t-test), and significantly to the right for the Δ⁻ condition (mean ± SE = 1.10° ± 0.10°; p = 5.4 × 10⁻²³, paired t-test). Thus, consistent with our previous study, both cues shifted (in opposite directions) to reduce cue conflict.

Comparing the vestibular vs. visual shift magnitudes (pooled by flipping the vestibular and visual shift signs in the Δ⁻ and Δ⁺ conditions, respectively) demonstrated significantly larger vestibular vs. visual shifts (1.43° ± 0.09° and 0.91° ± 0.07°, respectively; p = 6.8 × 10⁻⁵, paired t-test). This result is also consistent with our previous study. Thus, the behavioral results from the original study (performed in the Angelaki laboratory) were replicated in these experiments (in the Chen laboratory) using a new set of monkeys, with simultaneous neuronal recording. In the following sections, we present how neuronal responses in areas MSTd, PIVC, and VIP (Figures 3, 4 and 5, respectively) were recalibrated in comparison to the perceptual shifts.

Figure 3 with 1 supplement see all

Download asset Open asset

Figure 3—source data 1

Individual monkey summary statistics for dorsal medial superior temporal (MSTd) correlations.

https://cdn.elifesciences.org/articles/82895/elife-82895-fig3-data1-v1.doc

Download elife-82895-fig3-data1-v1.doc

Figure 3—source data 2

Comparison of pooled model (PM) and linear mixed model (LMM) for dorsal medial superior temporal (MSTd).

https://cdn.elifesciences.org/articles/82895/elife-82895-fig3-data2-v1.docx

Download elife-82895-fig3-data2-v1.docx

Figure 4

Download asset Open asset

Figure 4—source data 1

Individual monkey summary statistics for parietoinsular vestibular cortex (PIVC) correlations.

https://cdn.elifesciences.org/articles/82895/elife-82895-fig4-data1-v1.doc

Download elife-82895-fig4-data1-v1.doc

Figure 4—source data 2

Comparison of pooled model (PM) and linear mixed model (LMM) for parietoinsular vestibular cortex (PIVC).

https://cdn.elifesciences.org/articles/82895/elife-82895-fig4-data2-v1.docx

Download elife-82895-fig4-data2-v1.docx

Figure 5 with 1 supplement see all

Download asset Open asset

Figure 5—source data 1

Individual monkey summary statistics for ventral intraparietal (VIP) correlations.

https://cdn.elifesciences.org/articles/82895/elife-82895-fig5-data1-v1.doc

Download elife-82895-fig5-data1-v1.doc

Figure 5—source data 2

Comparison of pooled model (PM) and linear mixed model (LMM) for ventral intraparietal (VIP).

https://cdn.elifesciences.org/articles/82895/elife-82895-fig5-data2-v1.docx

Download elife-82895-fig5-data2-v1.docx

Temporal evolution of the correlation between neuronal and perceptual shifts

The neurometric curves in Figures 3—5 were calculated using mean FRs averaged across the stimulus duration. But the self-motion stimuli generated by the platform and optic flow followed a specific dynamic time course, specifically, a Gaussian velocity profile and correspondingly a biphasic acceleration profile (see bottom row, Figure 6). Therefore, we further examined whether the correlations between neurometric and perceptual shifts depend on the time point within the stimulus interval.

Figure 6

Download asset Open asset

For MSTd neurons, positive correlations (between neuronal and perceptual shifts) were seen for both vestibular and visual cues during the stimulus (Figure 6A). Correlations increased toward the middle of the stimulus, and dropped off rapidly at the end of the stimulus. Significant correlations (blue and red asterisk markers for vestibular and visual cues, respectively) were only seen around the middle of the stimulus. Thus neural recalibration in MSTd (in accordance with behavioral recalibration) could reflect the velocity responses.

For PIVC neurons, positive correlations (between neuronal and perceptual shifts) were seen only for vestibular cues, during the stimulus (upper panel in Figure 6B). Like MSTd, the vestibular correlations seemed to follow the velocity profile of the stimulus, with significant values around the middle of the stimulus. Correlations in the visual condition were very weak and not significant (middle panel in Figure 6B).

A very different profile was seen in VIP. Firstly, as described above, correlations between neuronal and perceptual recalibration were positive for the vestibular cue (upper panel in Figure 6C) and negative for the visual cue (middle panel in Figure 6C). Furthermore, the time course of these correlations was different in VIP: they increased in size gradually (positively for vestibular and negatively for visual), reaching a maximum around the middle of the stimulus epoch (the velocity peak), but significant correlations were found for time intervals beyond the end of the stimulus. This pattern is in line with sustained neuronal activity described previously for VIP. However, here this sustained activity correlated with subsequent vestibular choices, and was contrary to visual choices.

VIP choice signals are reduced after cross-modal recalibration

Previous studies have found that neuronal responses in VIP are strongly influenced (sometimes even dominated) by choice signals (Chen et al., 2021; Zaidel et al., 2017). Hence our finding here, that neuronal tuning recalibrated contrary to perceptual shifts for the visual cue, was surprising and counterintuitive. We, therefore, wondered what happened to the strong choice signals for which VIP is renowned, which would predict that neuronal tuning (also for visual cues) would shift with behavior.

To visualize choice tuning for an example VIP neuron, we plotted ‘choice-conditioned’ tuning curves, namely, neuronal responses as a function of heading, separately for rightward and leftward choices (Figure 7). In the pre-recalibration block vestibular responses were strongly choice related (Figure 7A, left panel) – neuronal responses to the same heading stimulus were larger when followed by rightward (►, blue) vs. leftward (◄, cyan) choices (the blue line lies above the cyan line). After recalibration, the choice effect decreased (Figure 7A, right panel) – the choice-conditioned tuning curves were no longer separate. Similarly, visual responses were strongly choice-related pre-recalibration, and this decreased post-recalibration (Figure 7B). To quantify the choice (and sensory) components of neuronal activity, and to observe how these changed after recalibration, we applied a partial correlation analysis (Zaidel et al., 2017). For this example neuron, the partial choice correlation values (R_c, presented on the plots) were reduced both for vestibular and visual cues.

Figure 7

Download asset Open asset

Across our sample of VIP neurons, the choice partial correlations in the post-recalibration block were significantly reduced compared to the pre-recalibration block, for both vestibular and visual cues (p = 6.0 × 10⁻⁴ and p = 1.3 × 10⁻³, respectively, paired t-tests; Figure 8B). However, the heading partial correlations (R_h) did not differ significantly from pre- to post-recalibration, neither for vestibular not visual cues (p = 0.96 and p = 0.85, respectively, paired t-tests; Figure 8A). For these statistical comparisons and for plotting we used the squared partial correlations (which quantify the amount of unique variance explained by choice or heading). We did not observe any significant changes in partial correlations in areas PIVC and MSTd (Figure 8—figure supplement 1). Lastly, there was no evidence for differences between post- and pre-recalibration baseline FRs in any of the three areas (Figure 8—figure supplement 2). Thus, shifts in neuronal tuning are not explained by changes in baseline activity.

Figure 8 with 2 supplements see all

Download asset Open asset

This study provides the first demonstration of unsupervised (cross-modal) neuronal recalibration, in conjunction with perceptual recalibration, in single sessions. Single neurons from MSTd, PIVC, and VIP revealed clear but different patterns of recalibration. In MSTd, neuronal responses to vestibular and visual cues shifted – each according to their respective cues’ perceptual shifts. In PIVC, vestibular tuning similarly shifted in the same direction as vestibular perceptual shifts (the PIVC cells were not robustly tuned to visual stimuli). However, recalibration in VIP was notably different: both vestibular and visual neuronal tuning shifted in the direction of the vestibular perceptual shifts. Thus, visual neuronal tuning shifted, surprisingly, contrary to visual perceptual shifts. These results indicate that neuronal recalibration differs profoundly across multisensory cortical areas.

The data and analysis code for this study have been uploaded to Github and can be found at https://github.com/FuZengBio/Recalibration (copy archived at Zeng, 2022).

1. 1. Abdulkarim Z
   2. Hayatou Z
   3. Ehrsson HH

   (2021) Sustained rubber hand illusion after the end of visuotactile stimulation with a similar time course for the reduction of subjective ownership and proprioceptive drift

   Experimental Brain Research 239:3471–3486.

   https://doi.org/10.1007/s00221-021-06211-8

   • PubMed
   • Google Scholar
2. 1. Amedi A
   2. Jacobson G
   3. Hendler T
   4. Malach R
   5. Zohary E

   (2002) Convergence of visual and tactile shape processing in the human lateral occipital complex

   Cerebral Cortex 12:1202–1212.

   https://doi.org/10.1093/cercor/12.11.1202

   • PubMed
   • Google Scholar
3. 1. Atkins JE
   2. Jacobs RA
   3. Knill DC

   (2003) Experience-dependent visual cue recalibration based on discrepancies between visual and haptic percepts

   Vision Research 43:2603–2613.

   https://doi.org/10.1016/s0042-6989(03)00470-x

   • PubMed
   • Google Scholar
4. 1. Avillac M
   2. Olivier E
   3. Den S
   4. Ben Hamed S
   5. Duhamel JR

   (2004) Multisensory integration in multiple reference frames in the posterior parietal cortex

   Cognitive Processing 5:159–166.

   https://doi.org/10.1007/s10339-004-0021-3

   • Google Scholar
5. 1. Avillac M
   2. Denève S
   3. Olivier E
   4. Pouget A
   5. Duhamel J-R

   (2005) Reference frames for representing visual and tactile locations in parietal cortex

   Nature Neuroscience 8:941–949.

   https://doi.org/10.1038/nn1480

   • PubMed
   • Google Scholar
6. Book

   1. Bertelson P
   2. De Gelder B

   (2004) The psychology of multimodal perception

   In: Charles Spence, Jon Driver, editors. Crossmodal Space and Crossmodal Attention. Oxford University Press. pp. 141–177.

   https://doi.org/10.1093/acprof:oso/9780198524861.001.0001

   • Google Scholar
7. 1. Botvinick M
   2. Cohen J

   (1998) Rubber hands “feel” touch that eyes see

   Nature 391:756.

   https://doi.org/10.1038/35784

   • PubMed
   • Google Scholar
8. 1. Brainard MS
   2. Knudsen EI

   (1993) Experience-dependent plasticity in the inferior colliculus: A site for visual calibration of the neural representation of auditory space in the barn owl

   The Journal of Neuroscience 13:4589–4608.

   https://doi.org/10.1523/JNEUROSCI.13-11-04589.1993

   • PubMed
   • Google Scholar
9. 1. Bremmer F
   2. Klam F
   3. Duhamel JR
   4. Ben Hamed S
   5. Graf W

   (2002) Visual-Vestibular interactive responses in the macaque ventral intraparietal area (VIP)

   The European Journal of Neuroscience 16:1569–1586.

   https://doi.org/10.1046/j.1460-9568.2002.02206.x

   • PubMed
   • Google Scholar
10. 1. Britten KH
    2. van Wezel RJ

    (1998) Electrical microstimulation of cortical area MST biases heading perception in monkeys

    Nature Neuroscience 1:59–63.

    https://doi.org/10.1038/259

    • PubMed
    • Google Scholar
11. 1. Britten KH
    2. Van Wezel RJA

    (2002) Area MST and heading perception in macaque monkeys

    Cerebral Cortex 12:692–701.

    https://doi.org/10.1093/cercor/12.7.692

    • PubMed
    • Google Scholar
12. 1. Britten KH

    (2008) Mechanisms of self-motion perception

    Annual Review of Neuroscience 31:389–410.

    https://doi.org/10.1146/annurev.neuro.29.051605.112953

    • PubMed
    • Google Scholar
13. 1. Burge J
    2. Girshick AR
    3. Banks MS

    (2010) Visual-haptic adaptation is determined by relative reliability

    The Journal of Neuroscience 30:7714–7721.

    https://doi.org/10.1523/JNEUROSCI.6427-09.2010

    • PubMed
    • Google Scholar
14. 1. Bush G
    2. Luu P
    3. Posner MI

    (2000) Cognitive and emotional influences in anterior cingulate cortex

    Trends in Cognitive Sciences 4:215–222.

    https://doi.org/10.1016/s1364-6613(00)01483-2

    • PubMed
    • Google Scholar
15. 1. Butler JS
    2. Smith ST
    3. Campos JL
    4. Bülthoff HH

    (2010) Bayesian integration of visual and vestibular signals for heading

    Journal of Vision 10:23.

    https://doi.org/10.1167/10.11.23

    • PubMed
    • Google Scholar
16. 1. Butler JS
    2. Campos JL
    3. Bülthoff HH

    (2015) Optimal visual-vestibular integration under conditions of conflicting intersensory motion profiles

    Experimental Brain Research 233:587–597.

    https://doi.org/10.1007/s00221-014-4136-1

    • PubMed
    • Google Scholar
17. 1. Canon LK

    (1970) Intermodality inconsistency of input and directed attention as determinants of the nature of adaptation

    Journal of Experimental Psychology 84:141–147.

    https://doi.org/10.1037/h0028925

    • PubMed
    • Google Scholar
18. 1. Chen A
    2. DeAngelis GC
    3. Angelaki DE

    (2010) Macaque parieto-insular vestibular cortex: Responses to self-motion and optic flow

    The Journal of Neuroscience 30:3022–3042.

    https://doi.org/10.1523/JNEUROSCI.4029-09.2010

    • PubMed
    • Google Scholar
19. 1. Chen A
    2. DeAngelis GC
    3. Angelaki DE

    (2011a) A comparison of vestibular spatiotemporal tuning in macaque parietoinsular vestibular cortex, ventral intraparietal area, and medial superior temporal area

    The Journal of Neuroscience 31:3082–3094.

    https://doi.org/10.1523/JNEUROSCI.4476-10.2011

    • PubMed
    • Google Scholar
20. 1. Chen A
    2. DeAngelis GC
    3. Angelaki DE

    (2011b) Representation of vestibular and visual cues to self-motion in ventral intraparietal cortex

    The Journal of Neuroscience 31:12036–12052.

    https://doi.org/10.1523/JNEUROSCI.0395-11.2011

    • PubMed
    • Google Scholar
21. 1. Chen A
    2. DeAngelis GC
    3. Angelaki DE

    (2013a) Functional specializations of the ventral intraparietal area for multisensory heading discrimination

    The Journal of Neuroscience 33:3567–3581.

    https://doi.org/10.1523/JNEUROSCI.4522-12.2013

    • PubMed
    • Google Scholar
22. 1. Chen X
    2. DeAngelis GC
    3. Angelaki DE

    (2013b) Diverse spatial reference frames of vestibular signals in parietal cortex

    Neuron 80:1310–1321.

    https://doi.org/10.1016/j.neuron.2013.09.006

    • PubMed
    • Google Scholar
23. 1. Chen A
    2. Gu Y
    3. Liu S
    4. DeAngelis GC
    5. Angelaki DE

    (2016) Evidence for a causal contribution of macaque vestibular, but not intraparietal, cortex to heading perception

    The Journal of Neuroscience 36:3789–3798.

    https://doi.org/10.1523/JNEUROSCI.2485-15.2016

    • PubMed
    • Google Scholar
24. 1. Chen X
    2. DeAngelis GC
    3. Angelaki DE

    (2018) Flexible egocentric and allocentric representations of heading signals in parietal cortex

    PNAS 115:E3305–E3312.

    https://doi.org/10.1073/pnas.1715625115

    • PubMed
    • Google Scholar
25. 1. Chen A
    2. Zeng F
    3. DeAngelis GC
    4. Angelaki DE

    (2021) Dynamics of heading and choice-related signals in the parieto-insular vestibular cortex of macaque monkeys

    The Journal of Neuroscience 41:3254–3265.

    https://doi.org/10.1523/JNEUROSCI.2275-20.2021

    • PubMed
    • Google Scholar
26. 1. Colby CL
    2. Duhamel JR
    3. Goldberg ME

    (1993) Ventral intraparietal area of the macaque: Anatomic location and visual response properties

    Journal of Neurophysiology 69:902–914.

    https://doi.org/10.1152/jn.1993.69.3.902

    • PubMed
    • Google Scholar
27. 1. de Winkel KN
    2. Weesie J
    3. Werkhoven PJ
    4. Groen EL

    (2010) Integration of visual and inertial cues in perceived heading of self-motion

    Journal of Vision 10:1.

    https://doi.org/10.1167/10.12.1

    • PubMed
    • Google Scholar
28. 1. Dokka K
    2. DeAngelis GC
    3. Angelaki DE

    (2015) Multisensory integration of visual and vestibular signals improves heading discrimination in the presence of a moving object

    The Journal of Neuroscience 35:13599–13607.

    https://doi.org/10.1523/JNEUROSCI.2267-15.2015

    • PubMed
    • Google Scholar
29. 1. Duffy CJ
    2. Wurtz RH

    (1995) Response of monkey MST neurons to optic flow stimuli with shifted centers of motion

    The Journal of Neuroscience 15:5192–5208.

    https://doi.org/10.1523/JNEUROSCI.15-07-05192.1995

    • PubMed
    • Google Scholar
30. 1. Duhamel JR
    2. Colby CL
    3. Goldberg ME

    (1998) Ventral intraparietal area of the macaque: congruent visual and somatic response properties

    Journal of Neurophysiology 79:126–136.

    https://doi.org/10.1152/jn.1998.79.1.126

    • PubMed
    • Google Scholar
31. 1. Fetsch CR
    2. Turner AH
    3. DeAngelis GC
    4. Angelaki DE

    (2009) Dynamic reweighting of visual and vestibular cues during self-motion perception

    The Journal of Neuroscience 29:15601–15612.

    https://doi.org/10.1523/JNEUROSCI.2574-09.2009

    • PubMed
    • Google Scholar
32. 1. Fetsch CR
    2. Pouget A
    3. DeAngelis GC
    4. Angelaki DE

    (2011) Neural correlates of reliability-based cue weighting during multisensory integration

    Nature Neuroscience 15:146–154.

    https://doi.org/10.1038/nn.2983

    • PubMed
    • Google Scholar
33. 1. Gu Y
    2. Watkins PV
    3. Angelaki DE
    4. DeAngelis GC

    (2006) Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area

    The Journal of Neuroscience 26:73–85.

    https://doi.org/10.1523/JNEUROSCI.2356-05.2006

    • PubMed
    • Google Scholar
34. 1. Gu Y
    2. DeAngelis GC
    3. Angelaki DE

    (2007) A functional link between area mstd and heading perception based on vestibular signals

    Nature Neuroscience 10:1038–1047.

    https://doi.org/10.1038/nn1935

    • PubMed
    • Google Scholar
35. 1. Gu Y
    2. Angelaki DE
    3. DeAngelis GC

    (2008) Neural correlates of multisensory cue integration in macaque mstd

    Nature Neuroscience 11:1201–1210.

    https://doi.org/10.1038/nn.2191

    • PubMed
    • Google Scholar
36. 1. Gu Y
    2. Fetsch CR
    3. Adeyemo B
    4. DeAngelis GC
    5. Angelaki DE

    (2010) Decoding of mstd population activity accounts for variations in the precision of heading perception

    Neuron 66:596–609.

    https://doi.org/10.1016/j.neuron.2010.04.026

    • PubMed
    • Google Scholar
37. 1. Gu Y
    2. DeAngelis GC
    3. Angelaki DE

    (2012) Causal links between dorsal medial superior temporal area neurons and multisensory heading perception

    The Journal of Neuroscience 32:2299–2313.

    https://doi.org/10.1523/JNEUROSCI.5154-11.2012

    • PubMed
    • Google Scholar
38. 1. Gu Y

    (2018) Vestibular signals in primate cortex for self-motion perception

    Current Opinion in Neurobiology 52:10–17.

    https://doi.org/10.1016/j.conb.2018.04.004

    • PubMed
    • Google Scholar
39. 1. Held R

    (1961) Exposure-history as a factor in maintaining stability of perception and coordination

    The Journal of Nervous and Mental Disease 132:26–32.

    https://doi.org/10.1097/00005053-196101000-00005

    • PubMed
    • Google Scholar
40. 1. Holroyd CB
    2. Coles MGH

    (2002) The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity

    Psychological Review 109:679–709.

    https://doi.org/10.1037/0033-295X.109.4.679

    • PubMed
    • Google Scholar
41. 1. Kennett S
    2. Taylor-Clarke M
    3. Haggard P

    (2001) Noninformative vision improves the spatial resolution of touch in humans

    Current Biology 11:1188–1191.

    https://doi.org/10.1016/s0960-9822(01)00327-x

    • PubMed
    • Google Scholar
42. 1. Knudsen EI
    2. Brainard MS

    (1991) Visual instruction of the neural map of auditory space in the developing optic tectum

    Science 253:85–87.

    https://doi.org/10.1126/science.2063209

    • PubMed
    • Google Scholar
43. 1. Knudsen EI

    (2002) Instructed learning in the auditory localization pathway of the barn owl

    Nature 417:322–328.

    https://doi.org/10.1038/417322a

    • PubMed
    • Google Scholar
44. 1. Kramer A
    2. Röder B
    3. Bruns P

    (2019) Feedback modulates audio-visual spatial recalibration

    Frontiers in Integrative Neuroscience 13:74.

    https://doi.org/10.3389/fnint.2019.00074

    • PubMed
    • Google Scholar
45. 1. Lewald J

    (2002) Rapid adaptation to auditory-visual spatial disparity

    Learning & Memory 9:268–278.

    https://doi.org/10.1101/lm.51402

    • PubMed
    • Google Scholar
46. 1. Linkenhoker BA
    2. Knudsen EI

    (2002) Incremental training increases the plasticity of the auditory space map in adult barn owls

    Nature 419:293–296.

    https://doi.org/10.1038/nature01002

    • PubMed
    • Google Scholar
47. 1. Markov DA
    2. Petrucco L
    3. Kist AM
    4. Portugues R

    (2021) A cerebellar internal model calibrates a feedback controller involved in sensorimotor control

    Nature Communications 12:6694.

    https://doi.org/10.1038/s41467-021-26988-0

    • PubMed
    • Google Scholar
48. 1. Maunsell JH
    2. van Essen DC

    (1983) The connections of the middle temporal visual area (MT) and their relationship to a cortical hierarchy in the macaque monkey

    The Journal of Neuroscience 3:2563–2586.

    https://doi.org/10.1523/JNEUROSCI.03-12-02563.1983

    • PubMed
    • Google Scholar
49. 1. Oman CM

    (1990) Motion sickness: A synthesis and evaluation of the sensory conflict theory

    Canadian Journal of Physiology and Pharmacology 68:294–303.

    https://doi.org/10.1139/y90-044

    • PubMed
    • Google Scholar
50. 1. Park H
    2. Kayser C

    (2021) The neurophysiological basis of the trial-wise and cumulative ventriloquism aftereffects

    The Journal of Neuroscience 41:1068–1079.

    https://doi.org/10.1523/JNEUROSCI.2091-20.2020

    • PubMed
    • Google Scholar
51. 1. Radeau M
    2. Bertelson P

    (1974) The after-effects of ventriloquism

    The Quarterly Journal of Experimental Psychology 26:63–71.

    https://doi.org/10.1080/14640747408400388

    • PubMed
    • Google Scholar
52. Book

    1. Reason JT
    2. Brand JJ

    (1975)

    Motion Sickness

    Cambridge, Massachusetts: Academic Press.

    • Google Scholar
53. 1. Recanzone GH

    (1998) Rapidly induced auditory plasticity: The ventriloquism aftereffect

    PNAS 95:869–875.

    https://doi.org/10.1073/pnas.95.3.869

    • PubMed
    • Google Scholar
54. 1. Rock I
    2. Victor J

    (1964) VISION and touch: An experimentally created conflict between the two senses

    Science 143:594–596.

    https://doi.org/10.1126/science.143.3606.594

    • PubMed
    • Google Scholar
55. 1. Rondi-Reig L
    2. Paradis AL
    3. Lefort JM
    4. Babayan BM
    5. Tobin C

    (2014) How the cerebellum may monitor sensory information for spatial representation

    Frontiers in Systems Neuroscience 8:205.

    https://doi.org/10.3389/fnsys.2014.00205

    • PubMed
    • Google Scholar
56. 1. Schlack A
    2. Hoffmann KP
    3. Bremmer F

    (2002) Interaction of linear vestibular and visual stimulation in the macaque ventral intraparietal area (VIP)

    The European Journal of Neuroscience 16:1877–1886.

    https://doi.org/10.1046/j.1460-9568.2002.02251.x

    • PubMed
    • Google Scholar
57. 1. Schlack A
    2. Sterbing-D’Angelo SJ
    3. Hartung K
    4. Hoffmann K-P
    5. Bremmer F

    (2005) Multisensory space representations in the macaque ventral intraparietal area

    The Journal of Neuroscience 25:4616–4625.

    https://doi.org/10.1523/JNEUROSCI.0455-05.2005

    • PubMed
    • Google Scholar
58. 1. Schroeder CE
    2. Foxe JJ

    (2002) The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex

    Brain Research. Cognitive Brain Research 14:187–198.

    https://doi.org/10.1016/s0926-6410(02)00073-3

    • PubMed
    • Google Scholar
59. 1. Sereno MI
    2. Huang RS

    (2014) Multisensory maps in parietal cortex

    Current Opinion in Neurobiology 24:39–46.

    https://doi.org/10.1016/j.conb.2013.08.014

    • PubMed
    • Google Scholar
60. 1. Shupak A
    2. Gordon CR

    (2006)

    Motion sickness: Advances in pathogenesis, prediction, prevention, and treatment

    Aviation, Space, and Environmental Medicine 77:1213–1223.

    • PubMed
    • Google Scholar
61. 1. Stein BE
    2. Stanford TR
    3. Rowland BA

    (2014) Development of multisensory integration from the perspective of the individual neuron

    Nature Reviews. Neuroscience 15:520–535.

    https://doi.org/10.1038/nrn3742

    • PubMed
    • Google Scholar
62. 1. Thériault R
    2. Landry M
    3. Raz A

    (2022) The rubber hand illusion: Top-down attention modulates embodiment

    Quarterly Journal of Experimental Psychology 75:2129–2148.

    https://doi.org/10.1177/17470218221078858

    • PubMed
    • Google Scholar
63. 1. Tsakiris M
    2. Haggard P

    (2005) The rubber hand illusion revisited: Visuotactile integration and self-attribution

    Journal of Experimental Psychology. Human Perception and Performance 31:80–91.

    https://doi.org/10.1037/0096-1523.31.1.80

    • PubMed
    • Google Scholar
64. 1. Ungerleider LG
    2. Desimone R

    (1986) Cortical connections of visual area MT in the macaque

    The Journal of Comparative Neurology 248:190–222.

    https://doi.org/10.1002/cne.902480204

    • PubMed
    • Google Scholar
65. 1. van Beers RJ
    2. Wolpert DM
    3. Haggard P

    (2002) When feeling is more important than seeing in sensorimotor adaptation

    Current Biology 12:834–837.

    https://doi.org/10.1016/s0960-9822(02)00836-9

    • PubMed
    • Google Scholar
66. 1. Vrieze SI

    (2012) Model selection and psychological theory: A discussion of the differences between the akaike information criterion (AIc) and the Bayesian information criterion (Bic)

    Psychological Methods 17:228–243.

    https://doi.org/10.1037/a0027127

    • PubMed
    • Google Scholar
67. 1. Warren WH
    2. Morris MW
    3. Kalish M

    (1988) Perception of translational heading from optical flow

    Journal of Experimental Psychology. Human Perception and Performance 14:646–660.

    https://doi.org/10.1037//0096-1523.14.4.646

    • PubMed
    • Google Scholar
68. 1. Watson DM
    2. Akeroyd MA
    3. Roach NW
    4. Webb BS

    (2021) Multiple spatial reference frames underpin perceptual recalibration to audio-visual discrepancies

    PLOS ONE 16:e0251827.

    https://doi.org/10.1371/journal.pone.0251827

    • PubMed
    • Google Scholar
69. 1. Webster MA

    (2012) Evolving concepts of sensory adaptation

    F1000 Biology Reports 4:21.

    https://doi.org/10.3410/B4-21

    • PubMed
    • Google Scholar
70. 1. Wurtz RH
    2. Duffy CJ

    (1992) Neuronal correlates of optic flow stimulation

    Annals of the New York Academy of Sciences 656:205–219.

    https://doi.org/10.1111/j.1749-6632.1992.tb25210.x

    • PubMed
    • Google Scholar
71. 1. Yu X
    2. Gu Y

    (2018) Probing sensory readout via combined choice-correlation measures and microstimulation perturbation

    Neuron 100:715–727.

    https://doi.org/10.1016/j.neuron.2018.08.034

    • PubMed
    • Google Scholar
72. 1. Zaidel A
    2. Turner AH
    3. Angelaki DE

    (2011) Multisensory calibration is independent of cue reliability

    The Journal of Neuroscience 31:13949–13962.

    https://doi.org/10.1523/JNEUROSCI.2732-11.2011

    • PubMed
    • Google Scholar
73. 1. Zaidel A
    2. Ma WJ
    3. Angelaki DE

    (2013) Supervised calibration relies on the multisensory percept

    Neuron 80:1544–1557.

    https://doi.org/10.1016/j.neuron.2013.09.026

    • PubMed
    • Google Scholar
74. 1. Zaidel A
    2. DeAngelis GC
    3. Angelaki DE

    (2017) Decoupled choice-driven and stimulus-related activity in parietal neurons may be misrepresented by choice probabilities

    Nature Communications 8:715.

    https://doi.org/10.1038/s41467-017-00766-3

    • PubMed
    • Google Scholar
75. 1. Zaidel A
    2. Laurens J
    3. DeAngelis GC
    4. Angelaki DE

    (2021) Supervised multisensory calibration signals are evident in VIP but not mstd

    The Journal of Neuroscience 41:10108–10119.

    https://doi.org/10.1523/JNEUROSCI.0135-21.2021

    • PubMed
    • Google Scholar
76. Software

    1. Zeng F

    (2022) Recalibration, version swh:1:rev:b0ed5b1149e31138787adb56e323dc24be678961

    Software Heritage.

    https://archive.softwareheritage.org/swh:1:dir:45cdf7288075bf2f9d2fe4b7d4aed210f7ae2e5f;origin=https://github.com/FuZengBio/Recalibration;visit=swh:1:snp:15baeb811aba265d34808d53aea1f26ef73865ec;anchor=swh:1:rev:b0ed5b1149e31138787adb56e323dc24be678961
77. 1. Zhang T
    2. Heuer HW
    3. Britten KH

    (2004) Parietal area VIP neuronal responses to heading stimuli are encoded in head-centered coordinates

    Neuron 42:993–1001.

    https://doi.org/10.1016/j.neuron.2004.06.008

    • PubMed
    • Google Scholar
78. 1. Zhao B
    2. Zhang Y
    3. Chen A

    (2021) Encoding of vestibular and optic flow cues to self-motion in the posterior superior temporal polysensory area

    The Journal of Physiology 599:3937–3954.

    https://doi.org/10.1113/JP281913

    • PubMed
    • Google Scholar
79. 1. Zierul B
    2. Röder B
    3. Tempelmann C
    4. Bruns P
    5. Noesselt T

    (2017) The role of auditory cortex in the spatial ventriloquism aftereffect

    NeuroImage 162:257–268.

    https://doi.org/10.1016/j.neuroimage.2017.09.002

    • PubMed
    • Google Scholar

Decision letter after peer review:

Thank you for submitting your article "Contrary neuronal recalibration in different multisensory cortical areas" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Michael Frank as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Umberto Olcese (Reviewer #2); Robbe Goris (Reviewer #3).

The reviewers' assessment of the work is overall quite positive; congratulations on an interesting and potentially impactful study. The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

1) Please report summary statistics (means and correlation coefficients) and p values for individual animals, for the data in Figure 2C, 3B, 4B, and 5B. This is not to say that the results must be significant at the individual animal level in order to support the study's main conclusions; we are all aware of the practicalities and accepted conventions of the field. Yet we feel it is important to be up front about this limitation, if indeed some of the individual monkey results fail to reach significance.

To be clear, the reasoning behind this request is that repeated observations from the same subject/condition are not independent of each other, thus correlations calculated from pooled data might obscure, inflate, or even reverse the true relationship between the variables (e.g., Simpson's Paradox).

Apart from individual subject analyses, another approach (indeed our strong recommendation) is to perform a hierarchical analysis. There are multiple options for this; probably the easiest one is to calculate linear mixed regression models (LMM) with one variable as predictor, the other as outcome and no intercept. This can be done in the frequentist way using (for example) the lmer package in R, or in a Bayesian way using Stan, which even as an automatized module for GLMMs. Partial correlations can be achieved by adding the variable that is partialized out as a predictor.

In summary, the authors may choose one (or more) of the three mentioned approaches to enhance statistical rigor: (1) separate t-tests and correlations for each animal and condition, (2) frequentist hierarchical linear models, and (3) Bayesian hierarchical linear models. It is worth pointing out that Bayesian approaches have advantages over frequentist methods, for instance quantifying the evidence for the null hypothesis (and thus better evaluating negative results such as Figure 4B-right).

2) Please consider applying a goodness-of-fit criterion to the neurometric curves before inclusion of their PSEs in the neuron-behavior correlation analyses – AND/OR evaluate the reliability of the PSEs using standard error obtained from the fitting procedure, or a bootstrap-based confidence interval. We would not require individual neuron shifts (δ-PSE) to be significant according to such SEs (i.e., through error propagation), but a reanalysis after removing particularly poor fits seems appropriate. The criterion to use for this is difficult for us to specify and thus can use your best judgment.

3) Depending on how the first two points are handled, and the outcome thereof, it may be necessary to scrutinize the role of outliers in generating the correlation coefficients and p values obtained. For instance, is the correlation of Figure 3B-left still significant without the upper four points, and 3B-right without the rightmost three points? A hierarchical analysis and/or neurometric goodness-of-fit criterion could reduce the role of outliers, in which case no formal outlier correction or other way to address this is needed.

Reviewer #1 (Recommendations for the authors):

The results are really interesting, yet, the manuscript in its current form needs revisions along two dimensions, 1) data analysis and 2) writing.

Methods

I am aware that basically all analyses in this manuscript have been used in published papers. The problems outlined below hold for those papers, too, sorry. Bad (or good) luck having this time a reviewer from another field.

Correlations derived from data that includes multiple repetitions per subject (and condition) require a hierarchical analysis because repeated observations from the same observer are not independent of each other. Subject A might on average have higher shifts (behavioral and neurometric) than subject B. In that case, a non-hierarchical analysis might return a significant positive correlation even if within subject A negative behavioral shifts are associated with positive neurometric shifts and vice versa. There are multiple ways to account for intra-subject correlations, probably the easiest one is to calculate linear mixed regression models (LMM) with one variable as predictor, the other as outcome and no intercept. This can be done in the frequentist way using the lmer package in R and in a Bayesian way using Stan which even as an automatized module for GLMMs. Partial correlations can be achieved by adding the variable that is partialized out as a predictor. Of course, it is also possible to just analyze each subject separately.

The same holds for any comparisons of mean values (across heading offsets during the recalibration phase or across modalities). For example, a t-test that includes perceptual shifts from all sessions and all monkeys is not valid as the values from a single monkey are correlated. Again, there are several ways to account for this. For example, an LMM with cross-modal discrepancy as predictor (and a free intercept) or each monkey's data could be tested separately.

The procedure to derive neurometric curves, which are central to the study, should be explained better. In the main text and in the figures, it should be clarified that each data point shows the proportion of trials in which an ideal observer would make a rightwards choice given only the firing rates of the neurons (and assumed anti-neurons). In the methods section, it should be explained in detail how the ROC curve is derived. Readers unfamiliar with the method should understand that the ROC incorporates decisions for a range of decision thresholds and that the area under the ROC curve (AUROC, not ROC value) corresponds to a theoretical observer's ability to discriminate between leftwards and rightwards motion given firing rates across repetitions of a specific motion direction. Finally, those AUROC values (ranging from 0.5 to 1) are mapped onto the probability of a rightwards choice (ranging from 0 to 1) given a heading direction.

The use of PSEs for the correlational analysis should be conditional on the goodness of fit of the sigmoid and the SD should fall within a reasonable range. A PSE derived from an ill-fitting or very flat curve has no informational value and is likely to be extreme, which in turn is huge a problem for any correlational analysis.

The methods section states that only neurons tuned to heading directions as indicated by cues in a specific modality were included in the neurometric analysis for that modality. If that exclusion criterion was applied, why are the neurometric curves for 'the other modality' so bad? I would have expected that neurons tuned to visual heading cues result in a good neurometric function for visual heading no matter where these neurons are located in the brain. In turn, I would have expected flat neurometric functions for visual heading direction in PIVC if also neurons tuned to vestibular heading information were included in the analysis. What am I missing?

Another instance of me probably not getting something: to identify neurons tuned for heading directions as cued by one of the two modalities, spike rates were regressed onto heading directions and only those neurons with a significant slope were included. I get that the range of tested headings is smaller than the width of a typical tuning curve and thus a linear regression makes more sense than trying to fit a full tuning curve. However, it seems to me that his selection method excludes neurons tuned to straight-ahead as their slope should be flat for a symmetric range of headings.

Regression and correlational analyses are extremely sensitive to outliers. To be sure that the results are robust, please repeat the analyses after outlier correction, e.g., based on the 1.5xIQR rule along each dimension or based on each data points influence on the regression (Cook's distance).

Looking at the currently depicted sigmoids, I suggest increasing the maximal value for the lapse rate. Even though it is not fully clear to me what a lapse rate of neurometric curve actually means, achieving a better fit for the curves seems essential given that the PSEs are at the center of the results.

The methods section provides information about the number of neurons per area included in the analyses in general. Additional information about the number of neurons per neurometric curve would be useful.

Please add in l. 698 how the baseline activity for each neuron was determined, e.g., based on a single interval at the beginning of the session or based on recordings in between trials.

Probably just a typo but in l. 689 Pearson's correlation has nothing to do with linear regression.

Writing

Generally, I find the manuscript to be well-written and organized. However, in its current form, the manuscript is geared towards a small audience, electrophysiologists familiar with most publications by the Angelaki and DeAngelis labs. A much wider audience could be reached by 1) phrasing more precisely and 2) providing more information, reasoning, and explanations.

The verb 'recalibrate' is often used in a way that doesn't match the literature and the way recalibration is thought of. Information cannot be recalibrated; cues cannot recalibrate actively; responses do not recalibrate (together). A system is recalibrated and then it interprets incoming information differently. In most instances, a simple replacement of recalibrated with "shifted" will help -- AND/OR include a statement early on defining these more colloquial uses of recalibrate (e.g., "we refer to this pattern of neural activity as 'recalibration'…)

A few examples for more common and precise phrasing:

l. 35 -> recalibrating itself based on information from …

l. 36 -> estimates for subsequently presented unisensory stimuli are shifted towards each other.

l. 25 -> the tuning of neural responses to … cues was shifted in the same direction as the monkeys' perceptual judgments of subsequently presented unisensory stimuli.

l. 43 -> tuning of vestibular neurons was shifted in the same direction as vestibular heading perception.

This issue is present throughout the manuscript but especially so in the abstract, in brief section, and the discussion. I doubt that someone not familiar with the literature can understand the abstract. There is a section in the discussion (l. 483f) that is written in a very abstract manner but phrased according to the literature, i.e., in accordance with the ways most people think about recalibration. Please adjust the rest of the text accordingly.

Comments in chronological order:

l. 57 Behavioral papers cited in a sentence about the neuronal basis of multisensory integration.

l. 76 Just saying that results exist is rather unusual. In a few words, what did the neuroimaging studies find?

l. 91 Burge et al. is not about heading perception but about visual-haptic slant perception.

l. 103 Again, the typical phrasing would be that perception was recalibrated as indicated by shifts in subsequent perceptual estimates.

l. 108f It would be easier for readers to learn about supervised recalibration in the discussion, as now they have to shift mentally from unsupervised to supervised and back to unsupervised recalibration.

l. 119 Why would diverging shifts in the tuning of neural mechanisms between cortical areas not be detectable when both perceptual estimates are shifted in the same direction?

l. 126 Isn't that what the neuroimaging studies showed? Changes in relatively early areas?

l. 140 "therefore we expected to see perceptual shifts resulting from unsupervised recalibration in MSTd and PIVC". That is a surprising claim, which needs further explanation as it implies that perceptual decisions are based on neural activity in these relatively early areas. Maybe leave that conclusion entirely to the discussion.

l. 145 The claim that VIP underlies cognitive processing will startle many. Why not simply say that VIP seems to be involved in perceptual decision making or higher order perceptual functions some of which yet have to be understood?

l. 161 Please explain that 1) during the recalibration block, the monkeys are exposed to visual-vestibular stimulus pairs with a consistent discrepancy in heading direction and that 2) in pre- and post-recalibration blocks the perception of heading direction indicated by unisensory stimuli is measured and that 3) recalibration effects are measured as the difference between pre- and post-recalibration results. The text should make sense to a reader who has never performed a recalibration study. No reader should be forced to read another paper just to understand the most crucial aspects of the current one!

l. 181f It does not seem necessary to describe the figure in detail in the text. Explanations of how to read a figure are best placed in its caption. Given that these are selected-curves of a single-subject in one of several sessions, the size of the shifts should not be compared or discussed in the text.

l. 200 At this point the reader has never heard that the monkeys underwent several sessions nor in which way the sessions differed from each other. Please add that information.

l. 202 Why force the reader to learn what δ + means in this manuscript? It is much better to just speak of 'sessions in which visual heading was rightwards of vestibular heading during the recalibration phase'. Please apply that thought throughout.

l. 232 What does 'tuning recalibrate with perceptual shifts' mean? A good advice I got from a book on scientific writing is to take sentences literally even when they describe scientific matter. One possible general phrasing would be that the neural tuning shifted in the same (opposite for VIP) direction as heading perception. A more results-oriented phrasing is that the neurometric functions shifted in the same direction as the psychometric functions for each modality. Please repair this throughout the text!

l. 240 This description of the neurometric analysis is not sufficient (see comment in the methods section). In addition, "PSEs were extracted similar to" is confusing, PSEs always correspond to the 50% point of a psychometric function (e.g., the mean of the Gaussian distribution). More importantly, it should be explained to readers that the PSE of a neurometric curve is the physical heading direction at which the chances to make a rightward judgment based on the firing rates of the neurons are fifty-fifty, i.e., straight-ahead according to the neuronal data.

l. 245 I think this is the first instance in which the term "behavioral shifts" is used instead of "perceptual shifts" but then it occurs consistently and is even present in the figures. In psychophysics, the term "behavioral shifts" would be used if there is any reason to suspect that the behavior does not correspond to the percepts, e.g., because participants show a response bias rather than a perceptual bias. If this might be the case, it should be discussed, otherwise please use perceptual.

l. 286 Again, please phrase more precisely.

l. 315f I like the analysis and the paragraph could function as a prototype for the subsequent paragraphs regarding its degree of abstraction and briefness. Yet, again more precise phrasing would be nice, e.g., neurons don't respond visually they exclusively respond when visual stimuli are presented.

l. 334f Again, in my view it is not necessary to describe figures and do so panel by panel.

l. 339f The last sentence of the paragraph does not parse.

The claim that neural recalibration follows the velocity profile of the stimulus is too strong and not correct as the figures show correlations not neural recalibration. For the earlier areas, the claim that the significant correlation between neurometric and psychometric shifts is driven by firing rates during the period of maximal stimulus velocity might be correct.

l. 350f Similarly, for VIP neurons it is also not necessary to describe the figure.

l. 356 The claim that the correlations between neurometric and psychometric shifts given firing rates recorded at the end of the stimulus presentation has nothing to do with choice behavior but reflects neuronal recalibration is not substantiated at this point, an anti-correlation is a strong relation. The conclusion might be drawn based on the last section of the results.

l. 364 "During recalibration" means during the recalibration phase but I think that is not what the authors mean because there are no behavioral choices during the recalibration phase. Please search the text for this phrasing, it probably is not adequate at other instances, too.

l. 412 Not sure if this is the first instance, but cross-sensory is not a word used in the literature, please use either multisensory or cross-modal or across the senses. Some authors will point out that multisensory should only be used in the case of perceptual fusion. Please replace the word throughout the text.

l. 412f This holds for the full Discussion: See above comments, the phrasing is very uncommon, especially the use of the verb 'recalibrate'. Almost all instances of 'recalibrated' should be replaced with 'changed' or 'shifted'. "Together with" means "shifts in same direction as perceptual shifts" and so on. It gets much better from line 478 on.

l. 422 Not sure if vision scientists would call MSTd a multisensory area.

l. 432f Why does 'unsupervised' imply changes in early areas? I wondered the same in the introduction. And how does that go together with the suggestion that the conflict is detected in ACC? It might be easiest to just refer to the neuroimaging studies or simply not make a claim.

l. 457f What is "individualized recalibration"? The literature uses "modality-specific". I fail to see what this section adds that is not in the previous section. Why would the yoking found in the supervised recalibration study for both perceptual and neuronal shifts predict uniformity in the neuronal shifts in a paradigm that leads to non-yoked perceptual shifts?

l. 499 Please phrase the conclusion as a possibility, as it remains unclear to which degree supervised and unsupervised recalibration correspond.

l. 513 'Shift in reference frame' does this refer to a change in the supramodal definition of straight-ahead based on the vestibular tuning in lower sensory areas? The idea that VIP is tuned in a vestibular reference frame fits with earlier studies investigating visual-tactile reference frames (e.g., Avillac et al., Sereno and Huang, and also Graziano recording in F4).

l. 531 what does "cross-sensory recalibration, vs. simply reflecting the recalibrated signals" mean? What are recalibrated signals and why are they different from cross-modal recalibration?

General writing guidelines:

All figures should be optimized for the outlet, which in the case of eLife is wide and short figures as the text is never set in two-columns.

All acronyms should be defined when they are used for the first time, e.g., point of subjective equivalence (PSE). It can be very helpful for readers to treat abstract, significance statement, main text, and methods as separate and define acronyms anew.

A number and its unit are separated by a space, e.g., 300 ms instead of 300ms. Please check this throughout the text including the figures.

Figure 1

A: A visualization of the optic flow stimuli (e.g., two subsequent frames or little arrows indicating the motion vectors) would be nice.

B: It should be indicated either in the figure or in the caption that δ was constant within a single session, but theta was varied within a session and could take on all values depicted in A.

The grey vector corresponds to the combined direction if both cues have the same reliability, which probably wasn't exactly the case given Figure 2A.

C: 'no choice' is misplaced.

All of them, please add a space between a number and its unit.

Figure 2

A,B please indicate the monkey and the session number.

A,B usually rightward choice ratio would be interpreted as the ratio of rightward to leftward choices, I assume the proportion of rightward responses is shown.

C indicate how the shift was calculated.

C I cannot see the error bars referred to in the caption.

C please indicate the distribution of each monkey to assure readers that the results hold within and across subjects (see comments on the statistical analysis).

Reviewer #2 (Recommendations for the authors):

Aside from the major comments outlined in the public review, I found that some figure legends should be expanded. For instance, in Figure 8 it is unclear what the empty and filled circles indicate, respectively. I would recommend the authors to check the manuscript carefully.

While in general the manuscript is well written, I found the "in brief" section rather difficult and not suitable for a broader audience. I would suggest rewriting the section accordingly.

Essential revisions:

1) Please report summary statistics (means and correlation coefficients) and p values for individual animals, for the data in Figure 2C, 3B, 4B, and 5B. This is not to say that the results must be significant at the individual animal level in order to support the study's main conclusions; we are all aware of the practicalities and accepted conventions of the field. Yet we feel it is important to be up front about this limitation, if indeed some of the individual monkey results fail to reach significance.

To be clear, the reasoning behind this request is that repeated observations from the same subject/condition are not independent of each other, thus correlations calculated from pooled data might obscure, inflate, or even reverse the true relationship between the variables (e.g., Simpson's Paradox).

Apart from individual subject analyses, another approach (indeed our strong recommendation) is to perform a hierarchical analysis. There are multiple options for this; probably the easiest one is to calculate linear mixed regression models (LMM) with one variable as predictor, the other as outcome and no intercept. This can be done in the frequentist way using (for example) the lmer package in R, or in a Bayesian way using Stan, which even as an automatized module for GLMMs. Partial correlations can be achieved by adding the variable that is partialized out as a predictor.

In summary, the authors may choose one (or more) of the three mentioned approaches to enhance statistical rigor: (1) separate t-tests and correlations for each animal and condition, (2) frequentist hierarchical linear models, and (3) Bayesian hierarchical linear models. It is worth pointing out that Bayesian approaches have advantages over frequentist methods, for instance quantifying the evidence for the null hypothesis (and thus better evaluating negative results such as Figure 4B-right).

Thank you for raising this important point. In response, we 1) added summary statistics for the individual monkeys, and 2) applied the linear mixed model (LMM) analysis to the data, with neuronal shifts as the dependent variable, perceptual shifts as the fixed predictor, and monkeys as a random effects grouping factor. Results from both of these approaches indicate that the findings are consistent across monkeys. We added these results as supplementary tables (see source data associated with each figure) to the revised manuscript. The LMM analyses (with monkeys as a random effects factor) did not provide better fits than the pooled model (PM) analyses (with pooled data across monkeys). These additional analyses support the main findings of the manuscript.

2) Please consider applying a goodness-of-fit criterion to the neurometric curves before inclusion of their PSEs in the neuron-behavior correlation analyses – AND/OR evaluate the reliability of the PSEs using standard error obtained from the fitting procedure, or a bootstrap-based confidence interval. We would not require individual neuron shifts (δ-PSE) to be significant according to such SEs (i.e., through error propagation), but a reanalysis after removing particularly poor fits seems appropriate. The criterion to use for this is difficult for us to specify and thus can use your best judgment.

Thank you for this valid suggestion. To quantify the reliability of each neurometric PSE we used the standard deviation (SD) of the bootstrapped PSEs obtained from the fitting procedure. We considered that this measure would be a more direct estimate of PSE reliability (the parameter of interest) vs. more general goodness-of-fit measures (such as pseudo-R²) which are influenced by other parameters (e.g., thresholds and lapse rates) that are less relevant for assessing PSE estimate reliability. We required that the SD of the neuron’s bootstrapped PSE was < 10°, in both the pre-and post-recalibration blocks. This cutoff was chosen empirically, based on the distribution of SD values across neurons (see Figure 3—figure supplement 1). This screening was applied in addition to (after) the initial screening for significant tuning (as described in the original manuscript). We have now updated the manuscript to say that the neurons were first screened for significant response tuning, and then the remaining neurons were screened for neurometric PSE reliability (please see the revised Methods subsection “Neuronal shifts”, paragraph 1).

For MSTd, the neurometric PSE reliability screening removed 9 neurons from the vestibular data and 9 neurons from the visual data. For PIVC, this removed 14 neurons from the vestibular data and 11 neurons from the visual data. For VIP, this removed 16 neurons from the vestibular data and 24 neurons from the visual data. We redid all the analyses including only the neurons that passed this additional screening (the results remained similar), updated the figures, and revised the manuscript accordingly.

3) Depending on how the first two points are handled, and the outcome thereof, it may be necessary to scrutinize the role of outliers in generating the correlation coefficients and p values obtained. For instance, is the correlation of Figure 3B-left still significant without the upper four points, and 3B-right without the rightmost three points? A hierarchical analysis and/or neurometric goodness-of-fit criterion could reduce the role of outliers, in which case no formal outlier correction or other way to address this is needed.

The robustness of the results was assessed with the systematic approaches described above (in response to the two previous comments). Namely: additional screening to remove neurons with low-reliability estimates for the neuronal PSE (see the response to Comment #2), and additional analysis (LMM) to account for the random effects of different monkeys (see the response to Comment #1). These specific data points passed this screening, and the results were robust to further analyses that took into account the random effects of monkeys. In addition, we tested the influence of outliers using ‘Cook's distance’ (Author response image 1) . Redoing the analyses after removing data points with Cook’s distances larger than three times the mean provided similar results as before (summarized in Author response table 1).

Author response image 1

Download asset Open asset

Reviewer #1 (Recommendations for the authors):

The results are really interesting, yet, the manuscript in its current form needs revisions along two dimensions, 1) data analysis and 2) writing.

Methods

I am aware that basically all analyses in this manuscript have been used in published papers. The problems outlined below hold for those papers, too, sorry. Bad (or good) luck having this time a reviewer from another field.

Correlations derived from data that includes multiple repetitions per subject (and condition) require a hierarchical analysis because repeated observations from the same observer are not independent of each other. Subject A might on average have higher shifts (behavioral and neurometric) than subject B. In that case, a non-hierarchical analysis might return a significant positive correlation even if within subject A negative behavioral shifts are associated with positive neurometric shifts and vice versa. There are multiple ways to account for intra-subject correlations, probably the easiest one is to calculate linear mixed regression models (LMM) with one variable as predictor, the other as outcome and no intercept. This can be done in the frequentist way using the lmer package in R and in a Bayesian way using Stan which even as an automatized module for GLMMs. Partial correlations can be achieved by adding the variable that is partialized out as a predictor. Of course, it is also possible to just analyze each subject separately.

Thank you for raising this valid point. In response, we: (1) calculated the individual monkey summary statistics, and report these in the revised manuscript (Figure 2–source data 1 presents the behavioral shift data; Figure 3–source data 1, Figure 4–source data 1, and Figure 5–source data 1 present the correlations between neuronal and behavioral shifts for MSTd, PIVC, and VIP, respectively). (2) We performed the linear mixed model (LMM) analysis, and present the results, and comparison thereof to the pooled model (PM) with pooled data across monkeys, in the respective supplementary tables (see source data). Please see our response to Essential Revision #1 (above) for further details.

The same holds for any comparisons of mean values (across heading offsets during the recalibration phase or across modalities). For example, a t-test that includes perceptual shifts from all sessions and all monkeys is not valid as the values from a single monkey are correlated. Again, there are several ways to account for this. For example, an LMM with cross-modal discrepancy as predictor (and a free intercept) or each monkey's data could be tested separately.

The procedure to derive neurometric curves, which are central to the study, should be explained better. In the main text and in the figures, it should be clarified that each data point shows the proportion of trials in which an ideal observer would make a rightwards choice given only the firing rates of the neurons (and assumed anti-neurons). In the methods section, it should be explained in detail how the ROC curve is derived. Readers unfamiliar with the method should understand that the ROC incorporates decisions for a range of decision thresholds and that the area under the ROC curve (AUROC, not ROC value) corresponds to a theoretical observer's ability to discriminate between leftwards and rightwards motion given firing rates across repetitions of a specific motion direction. Finally, those AUROC values (ranging from 0.5 to 1) are mapped onto the probability of a rightwards choice (ranging from 0 to 1) given a heading direction.

In response to this comment, we added more detailed explanations about the procedure to derive the neurometric curves (please see the revised Methods subsection “Neurometrics”, paragraphs 1-2). We also added to the figure legends (regarding neurometric curves) that each data point shows the proportion of trials in which an ideal observer would make a rightward choice given the firing rates of the neurons. We would like to clarify that for this calculation we did not compare neurons to “anti-neurons” (which would assume PSE = 0°). Rather, we z-scored the data in reference to the pre-calibration firing rates. Using a common reference (pre-recalibration mean, corresponding to z-score = 0) to normalize both pre- and post-recalibration data was needed to allow for and expose PSE shifts. Then we calculated ROC curves by moving a ‘criterion’ value from the minimum to the maximum z-score (in 100 steps), and plotting the probability that the z-scores exceeded the criterion vs. whether z-score = 0 (the pre-recalibration mean) exceeded that same criterion, or not (1 or 0, respectively). We have also added this clarification to the revised manuscript (see Methods subsection “Neurometrics”, paragraph 2).

The use of PSEs for the correlational analysis should be conditional on the goodness of fit of the sigmoid and the SD should fall within a reasonable range. A PSE derived from an ill-fitting or very flat curve has no informational value and is likely to be extreme, which in turn is huge a problem for any correlational analysis.

In response to this comment we further screened the neurons by calculating the SD of the neurometric PSE (from bootstrapped values) and excluded neurons with SDs > 10° (either pre- or post-recalibration). We considered that this would be a more direct measure of PSE reliability (the parameter of interest) vs. goodness-of-fit of the whole psychometric function, which could be affected by other parameters (such as thresholds and lapse rates). Please see our response to Essential Revision #2 (above) for further details.

The methods section states that only neurons tuned to heading directions as indicated by cues in a specific modality were included in the neurometric analysis for that modality. If that exclusion criterion was applied, why are the neurometric curves for 'the other modality' so bad? I would have expected that neurons tuned to visual heading cues result in a good neurometric function for visual heading no matter where these neurons are located in the brain. In turn, I would have expected flat neurometric functions for visual heading direction in PIVC if also neurons tuned to vestibular heading information were included in the analysis. What am I missing?

In the revised Methods subsection “Neuronal shifts”, paragraph 1, we now better explain the inclusion criteria for the different stages of analysis. First, we calculated neurometric functions for all recorded cells, and both modalities, whether or not the cell was significantly tuned. For the subsequent neuronal vs. behavioral shift analyses, we only included neurons that satisfied both the following criteria (the second criterion was added to the revised manuscript in response to Essential Revisions, Comment #2): i) significant tuning to the corresponding cue, and ii) reliable neurometric PSE estimates (SD < 10°, both in the pre- and post-recalibration blocks). The example neuron from PIVC in Figure 4 responded significantly to vestibular (but not visual) stimuli. Thus, although neurometric fits are presented for both cues in Figure 4, this PIVC neuron was included only in the vestibular (but not the visual) PSE correlation analysis.

Another instance of me probably not getting something: to identify neurons tuned for heading directions as cued by one of the two modalities, spike rates were regressed onto heading directions and only those neurons with a significant slope were included. I get that the range of tested headings is smaller than the width of a typical tuning curve and thus a linear regression makes more sense than trying to fit a full tuning curve. However, it seems to me that his selection method excludes neurons tuned to straight-ahead as their slope should be flat for a symmetric range of headings.

Indeed our selection criteria were selective for neurons that have sloped tuning around straight-ahead, and exclude neurons that could be tuned to forward motion stimuli (i.e., with a tuning preference straight-ahead). The reasons for this are: (1) neurons with sloped tuning straight-ahead are most informative for heading discrimination (large Fisher information, Gu et al., 2010). By contrast, neurons with a tuning preference to straight-ahead stimuli have relatively flat responses around straight-ahead (low Fisher information) and are thus less informative for heading discrimination. Accordingly, small shifts can be readily detected in neurons with sloped tuning (but not in those with flat tuning) around straight ahead. (2) In the cortical areas of interest in this study, a disproportionately large number of neurons have steep tuning slopes around straight ahead, presumably to support heading discrimination (Chen et al., 2011b; Gu et al., 2010). (3) The standard neurometric analysis wouldn’t work for neurons with a tuning preference to straight-ahead (rightward and leftward headings around straight-ahead would elicit ambiguous firing rates). Thus estimating tuning shifts in these neurons would require different (heterogeneous and more complex) models, and the results would be very noisy. Therefore, in this study, we focused on the prevalent neurons with sloped tuning around straight-ahead. We have added these points to the revised Methods subsection “Data analysis”, paragraph 3.

Regression and correlational analyses are extremely sensitive to outliers. To be sure that the results are robust, please repeat the analyses after outlier correction, e.g., based on the 1.5xIQR rule along each dimension or based on each data points influence on the regression (Cook's distance).

In this revision, we added: 1) summary statistics of the individual monkeys, and performed an LMM analysis (in response to Essential Revisions, Comment #1), and 2) we screened the neurons based on their PSE estimate reliability (in response to Essential Revisions, Comment #2). In addition, we also tested the influence of outliers using ‘Cook's distance’ (see details in response to Essential Revisions, Comment #3). Results from these analyses support the robustness of the results.

Looking at the currently depicted sigmoids, I suggest increasing the maximal value for the lapse rate. Even though it is not fully clear to me what a lapse rate of neurometric curve actually means, achieving a better fit for the curves seems essential given that the PSEs are at the center of the results.

We assessed the effect of allowing for a broader range of lapse rates on the neurometric fits, and PSEs in particular. Allowing for a broader range of lapse rates increased pseudo-R² values (expected when increasing the number of model parameters) and sometimes affected threshold estimates (thresholds could be under- or overestimated, depending on whether lapse rates are allowed or not, respectively). However, it had little influence on the PSE estimates. For example, the post-recalibration neurometric curve in Figure 3A (blue; redrawn in Author response image 2A) has a threshold = 17.8°. If larger lapse rates are allowed (Author response image 2B), this could provide an unreasonably low threshold estimate (threshold = 0.6°). Thus, allowing (or not) large lapse rates can drastically affect threshold estimates. By contrast, the post-recalibration neurometric PSE was 3.6° and 2.8° for the two curves, respectively. Hence, in general, the effect of lapse rates on PSE estimates is small. Because PSE (not threshold) is the parameter of interest in this study, we opted for a more conservative neuromeric fit (with fewer parameters, i.e. tighter lapse rates) and exclusively used the PSE (not threshold) estimates. In addition, we added screening of neuronal PSE estimate reliability (via bootstrapping) in the revised manuscript (please see details in response to Essential Revisions, Comment #2).

Author response image 2

Download asset Open asset

The methods section provides information about the number of neurons per area included in the analyses in general. Additional information about the number of neurons per neurometric curve would be useful.

We added this information in the Methods subsection “Neuronal shifts”, paragraph 1.

Please add in l. 698 how the baseline activity for each neuron was determined, e.g., based on a single interval at the beginning of the session or based on recordings in between trials.

Baseline FRs were calculated by taking the average FR in the 1 s window before stimulus onset. We have added this to the revised Methods, subsection “Data analysis”, paragraph 2.

Probably just a typo but in l. 689 Pearson's correlation has nothing to do with linear regression.

We have corrected this.

Writing

Generally, I find the manuscript to be well-written and organized. However, in its current form, the manuscript is geared towards a small audience, electrophysiologists familiar with most publications by the Angelaki and DeAngelis labs. A much wider audience could be reached by 1) phrasing more precisely and 2) providing more information, reasoning, and explanations.

In response to this comment and thanks to the comments below, we now provide more information and explanations, with improved phrasing in our revised manuscript.

The verb 'recalibrate' is often used in a way that doesn't match the literature and the way recalibration is thought of. Information cannot be recalibrated; cues cannot recalibrate actively; responses do not recalibrate (together). A system is recalibrated and then it interprets incoming information differently. In most instances, a simple replacement of recalibrated with "shifted" will help -- AND/OR include a statement early on defining these more colloquial uses of recalibrate (e.g., "we refer to this pattern of neural activity as 'recalibration'…)

In the revised manuscript we now use the term ‘recalibrate’ more carefully. This entailed replacing many instances of “recalibrated” with “shifted”.

A few examples for more common and precise phrasing:

l. 35 -> recalibrating itself based on information from …

Thank you for the improved phrasing. We have updated the text.

l. 36 -> estimates for subsequently presented unisensory stimuli are shifted towards each other.

Thank you for the improved phrasing. We have updated that sentence.

l. 25 -> the tuning of neural responses to … cues was shifted in the same direction as the monkeys' perceptual judgments of subsequently presented unisensory stimuli.

Thank you for the improved phrasing. We have updated that sentence.

l. 43 -> tuning of vestibular neurons was shifted in the same direction as vestibular heading perception.

This issue is present throughout the manuscript but especially so in the abstract, in brief section, and the discussion. I doubt that someone not familiar with the literature can understand the abstract. There is a section in the discussion (l. 483f) that is written in a very abstract manner but phrased according to the literature, i.e., in accordance with the ways most people think about recalibration. Please adjust the rest of the text accordingly.

Thanks for bringing this to our attention. We have updated the relevant text accordingly.

Comments in chronological order:

l. 57 Behavioral papers cited in a sentence about the neuronal basis of multisensory integration.

We updated those references

l. 76 Just saying that results exist is rather unusual. In a few words, what did the neuroimaging studies find?

We agree and have now added a brief explanation of these neuroimaging studies in the revised Introduction.

l. 91 Burge et al. is not about heading perception but about visual-haptic slant perception.

Thank you. We removed this reference from the paper.

l. 103 Again, the typical phrasing would be that perception was recalibrated as indicated by shifts in subsequent perceptual estimates.

Thanks. We have updated that sentence.

l. 108f It would be easier for readers to learn about supervised recalibration in the discussion, as now they have to shift mentally from unsupervised to supervised and back to unsupervised recalibration.

We agree and have moved this paragraph about supervised recalibration to the Discussion.

l. 119 Why would diverging shifts in the tuning of neural mechanisms between cortical areas not be detectable when both perceptual estimates are shifted in the same direction?

We now better clarify this point in the revised manuscript. In the supervised calibration paradigm, the overall shifts for both vestibular and visual cues were “yoked” in the same direction. Thus, we could not dissociate whether neuronal shifts for a particular cue (e.g., vestibular) follow the behavioral shifts for that cue (vestibular) or the other cue (visual). Both predict shifts in the same direction, and it would be difficult to dissociate these based on small differences in the expected shift magnitudes (because of noise). This (unsupervised) paradigm elicits behavioral shifts in opposite directions, and thus can more readily discern if the vestibular neurometrics shift with visual (rather than vestibular) behavioral shifts. We have added this explanation to the revised Discussion subsection “Modality-specific recalibration of vestibular and visual cues”, paragraph 3.

l. 126 Isn't that what the neuroimaging studies showed? Changes in relatively early areas?

Yes, we have updated the sentence by referencing neuroimaging studies that found auditory-visual recalibration in relatively early sensory areas (Amedi et al., 2002; Zierul et al., 2017).

l. 140 "therefore we expected to see perceptual shifts resulting from unsupervised recalibration in MSTd and PIVC". That is a surprising claim, which needs further explanation as it implies that perceptual decisions are based on neural activity in these relatively early areas. Maybe leave that conclusion entirely to the discussion.

We now justify this hypothesis based on the neuroimaging studies that found auditory-visual recalibration in relatively early sensory areas. Also, we further explain that because unsupervised plasticity is sensory-driven (occurs as a result of the cross-modal discrepancy, in the absence of overt feedback) we expected to observe neural correlates of recalibration in these lower-level sensory areas, and for these to propagate along the perceptual decision-making hierarchy.

l. 145 The claim that VIP underlies cognitive processing will startle many. Why not simply say that VIP seems to be involved in perceptual decision making or higher order perceptual functions some of which yet have to be understood?

We thank the reviewer for this refinement, and have updated that explanation accordingly.

l. 161 Please explain that 1) during the recalibration block, the monkeys are exposed to visual-vestibular stimulus pairs with a consistent discrepancy in heading direction and that 2) in pre- and post-recalibration blocks the perception of heading direction indicated by unisensory stimuli is measured and that 3) recalibration effects are measured as the difference between pre- and post-recalibration results. The text should make sense to a reader who has never performed a recalibration study. No reader should be forced to read another paper just to understand the most crucial aspects of the current one!

We agree and have now added more details to explain the experimental procedure.

l. 181f It does not seem necessary to describe the figure in detail in the text. Explanations of how to read a figure are best placed in its caption. Given that these are selected-curves of a single-subject in one of several sessions, the size of the shifts should not be compared or discussed in the text.

We have removed the unnecessary details from the text, and also removed the shift size comparison for the example plots. We also modified the figure legends.

l. 200 At this point the reader has never heard that the monkeys underwent several sessions nor in which way the sessions differed from each other. Please add that information.

Thank you. We have now added information (specifically the number of sessions, and how the sessions differed) so that the reader can better understand the results which follow (please see updated Results subsection “Vestibular and visual perceptual estimates shift toward each other”, paragraph 3).

l. 202 Why force the reader to learn what δ + means in this manuscript? It is much better to just speak of 'sessions in which visual heading was rightwards of vestibular heading during the recalibration phase'. Please apply that thought throughout.

We have now added (in response to the Reviewer’s previous comment) a better explanation regarding the two types of sessions, and the meaning of δ + and δ -. Because these appear numerously throughout the text, and there is not enough space in the figures to write explicitly 'sessions in which visual heading was rightwards of vestibular heading during the recalibration phase’ etc., we think that it is clearer to keep this nomenclature.

l. 232 What does 'tuning recalibrate with perceptual shifts' mean? A good advice I got from a book on scientific writing is to take sentences literally even when they describe scientific matter. One possible general phrasing would be that the neural tuning shifted in the same (opposite for VIP) direction as heading perception. A more results-oriented phrasing is that the neurometric functions shifted in the same direction as the psychometric functions for each modality. Please repair this throughout the text!

We have amended the language accordingly.

l. 240 This description of the neurometric analysis is not sufficient (see comment in the methods section). In addition, "PSEs were extracted similar to" is confusing, PSEs always correspond to the 50% point of a psychometric function (e.g., the mean of the Gaussian distribution). More importantly, it should be explained to readers that the PSE of a neurometric curve is the physical heading direction at which the chances to make a rightward judgment based on the firing rates of the neurons are fifty-fifty, i.e., straight-ahead according to the neuronal data.

We have updated the text accordingly.

l. 245 I think this is the first instance in which the term "behavioral shifts" is used instead of "perceptual shifts" but then it occurs consistently and is even present in the figures. In psychophysics, the term "behavioral shifts" would be used if there is any reason to suspect that the behavior does not correspond to the percepts, e.g., because participants show a response bias rather than a perceptual bias. If this might be the case, it should be discussed, otherwise please use perceptual.

We agree with the reviewer’s suggestion. We have replaced the term "behavioral shifts" with "perceptual shifts".

l. 286 Again, please phrase more precisely.

We now changed this subtitle to “Neuronal tuning in VIP to both vestibular and visual stimuli shifted according to vestibular perceptual shifts”.

l. 315f I like the analysis and the paragraph could function as a prototype for the subsequent paragraphs regarding its degree of abstraction and briefness. Yet, again more precise phrasing would be nice, e.g., neurons don't respond visually they exclusively respond when visual stimuli are presented.

We have revised the phrasing accordingly.

l. 334f Again, in my view it is not necessary to describe figures and do so panel by panel.

We have removed unnecessary descriptions of the figure from the main text.

l. 339f The last sentence of the paragraph does not parse.

The claim that neural recalibration follows the velocity profile of the stimulus is too strong and not correct as the figures show correlations not neural recalibration. For the earlier areas, the claim that the significant correlation between neurometric and psychometric shifts is driven by firing rates during the period of maximal stimulus velocity might be correct.

We agree with this point and have refined the explanation in line with the reviewer’s suggestion.

l. 350f Similarly, for VIP neurons it is also not necessary to describe the figure.

Thanks, we have refined this paragraph.

l. 356 The claim that the correlations between neurometric and psychometric shifts given firing rates recorded at the end of the stimulus presentation has nothing to do with choice behavior but reflects neuronal recalibration is not substantiated at this point, an anti-correlation is a strong relation. The conclusion might be drawn based on the last section of the results.

We have removed that sentence.

l. 364 "During recalibration" means during the recalibration phase but I think that is not what the authors mean because there are no behavioral choices during the recalibration phase. Please search the text for this phrasing, it probably is not adequate at other instances, too.

We agree with the reviewer’s suggestion. We have replaced the phrase "during recalibration" with "after recalibration" where appropriate in the text.

l. 412 Not sure if this is the first instance, but cross-sensory is not a word used in the literature, please use either multisensory or cross-modal or across the senses. Some authors will point out that multisensory should only be used in the case of perceptual fusion. Please replace the word throughout the text.

We agree with the reviewer’s suggestion. We have replaced the phrase "cross-sensory " with " cross-modal" throughout the text.

l. 412f This holds for the full Discussion: See above comments, the phrasing is very uncommon, especially the use of the verb 'recalibrate'. Almost all instances of 'recalibrated' should be replaced with 'changed' or 'shifted'. "Together with" means "shifts in same direction as perceptual shifts" and so on. It gets much better from line 478 on.

We agree with the reviewer’s suggestion. We replaced many instances of the word “recalibrate” with “shifted”. We also replaced phrases like “recalibrated together with the corresponding vestibular perceptual shifts” with “shifted in the same direction as vestibular perceptual shifts” etc.

l. 422 Not sure if vision scientists would call MSTd a multisensory area.

Although MSTd is part of the extra-striate visual cortex, it has been extensively shown to have multisensory (including vestibular) responses. We back up this claim in the manuscript with relevant references.

l. 432f Why does 'unsupervised' imply changes in early areas? I wondered the same in the introduction. And how does that go together with the suggestion that the conflict is detected in ACC? It might be easiest to just refer to the neuroimaging studies or simply not make a claim.

We now justify this hypothesis (in the revised Introduction) based on the neuroimaging studies, as recommended by the reviewer.

l. 457f What is "individualized recalibration"? The literature uses "modality-specific". I fail to see what this section adds that is not in the previous section. Why would the yoking found in the supervised recalibration study for both perceptual and neuronal shifts predict uniformity in the neuronal shifts in a paradigm that leads to non-yoked perceptual shifts?

We agree with the reviewer’s suggested phrasing and replaced "individualized recalibration" with "modality-specific recalibration". In the supervised calibration paradigm, the overall shifts for both vestibular and visual cues were “yoked” in the same direction. Thus, we could not dissociate whether neuronal shifts for a particular cue (e.g., vestibular) follow the behavioral shifts for that cue (vestibular) or the other cue (visual). Both predict shifts in the same direction, and it would be difficult to dissociate these based on the expected shift magnitudes (because of noise). This (unsupervised) paradigm elicits behavioral shifts in opposite directions, and thus can more readily discern if the vestibular neurometrics shift with visual (rather than vestibular) behavioral shifts.

l. 499 Please phrase the conclusion as a possibility, as it remains unclear to which degree supervised and unsupervised recalibration correspond.

We agree with the reviewer’s suggestion and have amended the language accordingly.

l. 513 'Shift in reference frame' does this refer to a change in the supramodal definition of straight-ahead based on the vestibular tuning in lower sensory areas? The idea that VIP is tuned in a vestibular reference frame fits with earlier studies investigating visual-tactile reference frames (e.g., Avillac et al., Sereno and Huang, and also Graziano recording in F4).

We do not know if this necessarily means “a change in the supramodal definition of straight-ahead based on the vestibular tuning in lower sensory areas”. We just point out that neuronal signals in VIP have been previously shown to be in head/body centered coordinates, and that the shifts we observe here are in line with that notion. We have clarified this section, and added these relevant references, in the revised Discussion subsection “Contrary recalibration in higher-level area VIP ” paragraph 3. Thank you.

l. 531 what does "cross-sensory recalibration, vs. simply reflecting the recalibrated signals" mean? What are recalibrated signals and why are they different from cross-modal recalibration?

By “cortical areas actively involved in cross-modal recalibration” we mean that they had an instrumental (causal) role in vestibular-visual recalibration. By “simply reflecting the recalibrated signals” we mean that they were not instrumental (causal) for recalibration, and thus only reflect signals that were recalibrated in/by other brain areas. We have now refined and better explained this idea in the revised Discussion.

General writing guidelines:

All figures should be optimized for the outlet, which in the case of eLife is wide and short figures as the text is never set in two-columns.

We have changed the layout of Figures 3, 4, and 5 accordingly.

All acronyms should be defined when they are used for the first time, e.g., point of subjective equivalence (PSE). It can be very helpful for readers to treat abstract, significance statement, main text, and methods as separate and define acronyms anew.

A number and its unit are separated by a space, e.g., 300 ms instead of 300ms. Please check this throughout the text including the figures.

We thank the reviewer for these suggestions, and have amended the text accordingly.

Figure 1

A: A visualization of the optic flow stimuli (e.g., two subsequent frames or little arrows indicating the motion vectors) would be nice.

We added a schematic (rightmost in Figure 1A) to visualize the optic flow.

B: It should be indicated either in the figure or in the caption that δ was constant within a single session, but theta was varied within a session and could take on all values depicted in A.

We added these points to the legend of Figure 1.

The grey vector corresponds to the combined direction if both cues have the same reliability, which probably wasn't exactly the case given Figure 2A.

The grey vector was not meant to reflect the perceived ‘combined cue’, rather, it is just a convention for defining the stimuli (visual and vestibular stimuli were offset ± 5° relative to this). We have clarified this in the legend.

C: 'no choice' is misplaced.

Thanks. We have corrected this.

All of them, please add a space between a number and its unit.

OK. We have updated accordingly.

Figure 2

A,B please indicate the monkey and the session number.

We added this information to the figure legend.

A,B usually rightward choice ratio would be interpreted as the ratio of rightward to leftward choices, I assume the proportion of rightward responses is shown.

Yes. We replaced "ratio" with "proportion" in the y-label.

C indicate how the shift was calculated.

We added to the legend how the shift was calculated.

C I cannot see the error bars referred to in the caption.

The error bars are indeed difficult to see because the SEs are small. To improve visibility, we reduced the triangle symbol size and made the error bars thicker.

C please indicate the distribution of each monkey to assure readers that the results hold within and across subjects (see comments on the statistical analysis).

We updated Figure 2C to display the distribution of each monkey (using different texture patterns). In addition, the summary statistics for each monkey are presented in Figure 2-source data 1.

Reviewer #2 (Recommendations for the authors):

Aside from the major comments outlined in the public review, I found that some figure legends should be expanded. For instance, in Figure 8 it is unclear what the empty and filled circles indicate, respectively. I would recommend the authors to check the manuscript carefully.

We thank the reviewer for bringing this to our attention. Filled and empty circles indicate significant and non-significant partial correlations respectively. We have added this and other details to the figure legends.

While in general the manuscript is well written, I found the "in brief" section rather difficult and not suitable for a broader audience. I would suggest rewriting the section accordingly.

We removed the "in brief" and “highlights” sections from the manuscript because these are not standard in eLife.

Author details

1. Fu Zeng

   Key Laboratory of Brain Functional Genomics (Ministry of Education), East China Normal University, Shanghai, China

   Contribution

   Data curation, Formal analysis, Validation, Visualization, Methodology, Writing - original draft

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0009-0000-6857-6485

2. Adam Zaidel

   Gonda Multidisciplinary Brain Research Center, Bar-Ilan University, Ramat Gan, Israel

   Contribution

   Conceptualization, Formal analysis, Supervision, Funding acquisition, Validation, Methodology, Writing - review and editing

   For correspondence

   adam.zaidel@biu.ac.il

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4405-8717

3. Aihua Chen

   Key Laboratory of Brain Functional Genomics (Ministry of Education), East China Normal University, Shanghai, China

   Contribution

   Conceptualization, Data curation, Formal analysis, Supervision, Funding acquisition, Investigation, Methodology, Writing - original draft, Project administration, Writing - review and editing

   For correspondence

   ahchen@brain.ecnu.edu.cn

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5066-2844

• 435

  Page views

• 58

  Downloads

• 1

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.