1. Neuroscience
Download icon

Pre-saccadic remapping relies on dynamics of spatial attention

  1. Martin Szinte  Is a corresponding author
  2. Donatas Jonikaitis
  3. Dragan Rangelov
  4. Heiner Deubel
  1. Vrije Universiteit, The Netherlands
  2. Howard Hughes Medical Institute, Stanford University School of Medicine, United States
  3. Queensland Brain Institute, The University of Queensland, Australia
  4. Ludwig-Maximilians-Universität München, Germany
Research Article
  • Cited 0
  • Views 580
  • Annotations
Cite this article as: eLife 2018;7:e37598 doi: 10.7554/eLife.37598

Abstract

Each saccade shifts the projections of the visual scene on the retina. It has been proposed that the receptive fields of neurons in oculomotor areas are predictively remapped to account for these shifts. While remapping of the whole visual scene seems prohibitively complex, selection by attention may limit these processes to a subset of attended locations. Because attentional selection consumes time, remapping of attended locations should evolve in time, too. In our study, we cued a spatial location by presenting an attention-capturing cue at different times before a saccade and constructed maps of attentional allocation across the visual field. We observed no remapping of attention when the cue appeared shortly before saccade. In contrast, when the cue appeared sufficiently early before saccade, attentional resources were reallocated precisely to the remapped location. Our results show that pre-saccadic remapping takes time to develop suggesting that it relies on the spatial and temporal dynamics of spatial attention.

https://doi.org/10.7554/eLife.37598.001

Introduction

Our eye movements shift the visual scene on our retinas. These shifts go largely unnoticed and do not prevent efficient interaction with objects surrounding us. It has been proposed that the visual system compensates for such shifts using a copy of the motor command (Sperry, 1950) to anticipate changes in the visual scene from the planned eye movement. Such an active mechanism could maintain an impression of space constancy and allow us to effectively interact with visual objects. However, we typically do not keep track of the whole visual scene (O'Regan et al., 1999; Rensink et al., 1997). Studies have proposed that such visual compensation could be restricted to salient or task-relevant objects, selected by spatial attention (Cavanagh et al., 2010; Rolfs and Szinte, 2016). At the behavioral level, this compensation could result in anticipatory deployment of spatial attention to the retinal location that a visual stimulus will occupy after the saccade (Jonikaitis and Theeuwes, 2013; Rolfs et al., 2011; Szinte et al., 2015; Szinte et al., 2016). Such anticipatory deployment could explain observations that attention is allocated at a spatial target location almost immediately after a saccade (Jonikaitis et al., 2013; Yao et al., 2016b).

At the neuronal level, these visual compensations have been described as a remapping of visual neuron receptive fields. Remapping triggers an anticipatory and, sometimes, pre-saccadic response of neurons in frontal eye fields (FEF), lateral intra-parietal area (LIP) and superior colliculus (SC) with receptive fields centered on the post-saccadic retinal location of the attended object (Duhamel et al., 1992; Sommer and Wurtz, 2006; Walker et al., 1995). Remapping can facilitate tracking of task-relevant objects across saccades and allow rapid comparison between pre- and post-saccadic visual inputs (Crapse and Sommer, 2012). However, this remapping hypothesis has been challenged with new data collected within the FEF (Chen et al., 2018; Zirnsak and Moore, 2014). These studies found that, before a saccade, neurons respond to stimuli presented near the saccade target rather than to stimuli presented at remapped locations of the recorded receptive field (RF). These results were later termed ‘convergent remapping’ towards the saccade target in dissociation of the ‘forward remapping’, which would be parallel to the saccade vector (Neupane et al., 2016a). They led to the proposal that convergent remapping could manifest behaviorally as a spatially unspecific spread of attention around the saccade target (Zirnsak and Moore, 2014). Remapping of spatial attention before saccades, as reported in behavioral studies, therefore could be reinterpreted as attentional spread between saccade target and remapped location (Jonikaitis et al., 2013; Rolfs et al., 2011; Szinte et al., 2015; Szinte et al., 2016). Such interpretation of the convergent remapping effects predicts that locations surrounding the saccade target by up to 10 degrees of visual angle (dva) would receive all attentional benefits before the eyes start to move. To date, there are no behavioral studies mapping pre-saccadic attention in sufficient detail to disambiguate whether attention converges towards the saccade target, or is remapped in parallel to the saccade target, as earlier behavioral work suggested.

We developed a protocol that allowed us to measure detailed maps of pre-saccadic attention, by measuring the orientation sensitivity at multiple locations while participants prepared a saccade (Figure 1). We observed that attention was allocated to the saccade target location and did not spread to the nearby positions, about 4.2 degrees of visual angle (dva apart. Next, we measured remapping of attention in the presence of a salient cue during a saccade task, manipulating the timing of the cue relative to the saccade. Our reasoning was that if remapping is an attentional process (Cavanagh et al., 2010; Rolfs and Szinte, 2016), it will take some time for the attention shift to occur (Ling and Carrasco, 2006; Müller and Rabbitt, 1989; Nakayama and Mackeben, 1989; Rolfs and Carrasco, 2012). Therefore, stimuli presented just before a saccade would not leave enough time for remapping to develop and to be observed before the saccade. On the other hand, stimuli presented early enough should be remapped before the saccade. Indeed, we found that when the cue appeared shortly before saccade onset, spatial attention was allocated at the cued location but not at its remapped location. In contrast, when the cue appeared sufficiently early before saccade onset, attentional resources that were initially drawn to the cued location were re-allocated to its remapped location (i.e. the retinal location it will occupy after the saccade).

Stimulus displays and stimulus eccentricity effects.

(A) Participants fixated on the fixation target (ft) and prepared a saccade towards the saccade target (st) presented either to the right or to the left of the fixation between 700 and 900 ms after the trial onset. On each trial, 12 visual streams (40 Hz flickering vertical Gabors and masks) were shown and in two out of the three trials a cue was flashed (50 ms) either above or below a virtual line between the fixation and the saccade targets (note that the stimuli here are sketched to increase their visibility, actual stimuli match those shown in the visual stream depiction). (B) The arrangement of visual streams can take several positions (see Materials and methods), to cover the whole display across trials. Participants reported the orientation of a discrimination target (dt), a tilted Gabor, presented within all trials at a time maximizing the occurrence of its offset within the 150 ms preceding the saccade. (C–D) The discrimination target was shown across trials at 32 different positions (see black dots) covering 24 dva horizontally and 18 dva vertically and including four main positions of interest (the fixation target: ft; the saccade target: st, the cue: cue; and the remapped location of the cue: remap.). The tilt of the discrimination target was titrated to yield comparable performance at differently cued eccentricities from the fixation target. We adjusted these tilts in a preliminary task made either while participants kept their eyes steady at the fixation target (C) peripheral remapping threshold task (see Materials and methods), or prepared a saccade (D) foveal remapping threshold task (see Materials and methods). The maps show dt tilt angles averaged across participants in these two threshold tasks.

https://doi.org/10.7554/eLife.37598.002

Results

We determined spatially detailed maps of attention before a saccade under two different conditions: first, when participants made a visually guided saccade, and second, when a transient peripheral stimulus, a cue, was additionally presented during its preparation. We assessed spatial attention by asking participants to report the orientation of a briefly presented tilted discrimination target (clockwise or counterclockwise tilted Gabor), embedded in a display of vertical distractor streams (vertical Gabors, Figure 1A–B). To ensure that the discrimination task could be solved correctly only if participants attended at a particular location, we first completed a threshold task in which participants fixated at the center of the screen. This threshold task was used to estimate the tilt angle of a cued discrimination target presented at different eccentricities from the fixation. We observed that to achieve comparable discrimination at different eccentricities, the discrimination target had to be tilted by 4.42 ± 0.86 ° (mean ± SEM), if presented at the fixation target. This tilt gradually increased with eccentricity, finally reaching 14.10 ± 1.40 ° at eccentricities between ~15.3 and~16.2 dva (see Figure 1C). We used these threshold tilt values at their respective eccentricities in the main saccade task.

We first verified that presentation of the discrimination target during saccade preparation did not disrupt eye movements. Such a disruption, as measured by saccade latency or accuracy, would suggest that the stimuli used to measure attention instead captured attention. For this we first determined whether eccentricity of the discrimination target affected saccade latency. Saccade latency was longer when the visual streams overlapped with the fixation and saccade targets (217.56 ± 3.77 ms) compared with when they didn’t overlap (186.00 ± 2.61 ms, p < 0.0001). This indicates that such difference resulted from the saccade target and fixation being less visible if they overlapped with the visual streams. Therefore, we separated the trials based on whether the fixation and saccade targets overlapped with the visual streams or not. Discrimination target eccentricity did not affect saccade latency on trials in which the fixation and saccade targets overlapped with visual streams. We did not observe a main effect of the discrimination target eccentricity (see Materials and methods for the definition of eccentricity), either for trials in which the fixation and saccade target overlapped with the visual streams (repeated measures ANOVA for four eccentricity groups used, F3,39 = 0.08, p = 0.9725, ηp2= 0.03 %) or for trials in which they didn’t (for four eccentricity groups used, F3,39 = 1.49, p = 0.2312, ηp2= 0.83 %). Note that from a pilot study, we expected to find such saccade latency costs when the targets overlapped with the visual streams. To compensate for these expected effects, on trials in which the visual streams overlapped with the fixation and the saccade target, we presented the discrimination target 25 ms later than when there was no overlap. This procedure ensured homogenous timing of the discrimination target relative to the saccade onset irrespective of the tested position. This also ensured that any spatially unspecific increase of the discrimination threshold before a saccade (Campbell & Wurtz, 1978) could not explain differences between our experimental conditions and that discrimination performance truly reflected sensitivity gathered at the same instant relative to the saccade onset. Next, we evaluated whether discrimination target eccentricity (i.e. the absolute distance between the saccade target and the saccade landing point) affected saccade accuracy, as would be evident if the target captured attention. Again, we didn’t find a main effect of the discrimination target eccentricity on the saccade accuracy (between five eccentricity groups used, F4,52 = 2.11, p = 0.0929, ηp2= 1.38 %). Altogether, these results show that presentation of the discrimination target did not disrupt saccade preparation, demonstrating that the stimuli we used to measure the allocation of attention did not directly interfere with its deployment.

Next, we measured the pre-saccadic allocation of attention. In all conditions, we analyzed performance to the presentation of discrimination targets within the 150 ms preceding the saccade. This procedure ensured that discrimination performance reflected the modulation of spatial attention over space rather than visual acuity and ensured that the same discrimination targets were matched across all conditions and trials. Trials with and without a cue were analyzed separately. On trials without a cue (Figure 2A–C), we found increased visual sensitivity (0.88 ± 0.05, normalized d’ and SEM, respectively) at the saccade target location relative to the average of all other tested positions (0.42 ± 0.04, p < 0.0001, Figure 2B), suggesting that attention shifted towards the saccade targets during saccade preparation. Further, we tested the spatial specificity of this effect by comparing visual sensitivity at the saccade target location with the average visual sensitivity at the four positions surrounding it. Attention at the saccade target clearly did not spread to the surrounding positions (0.40 ± 0.04, p < 0.0001 , Figure 2C), with sensitivity benefits being constrained to the immediate vicinity of the saccade target. Also, we did not observe a deployment of spatial attention to the fixation target (0.41 ± 0.05) compared with the average across all other positions (p = 0.8952).

Stimulus timing and sensitivity maps.

(A,D,G) Stimulus timing. Participants prepared a saccade at the offset of the fixation target (ft), which corresponded to the onset of the saccade target (st). In the 150 ms before the saccade, a discrimination target (dt) was briefly shown at one of the 32 possible positions. Then, no cue was shown (A), or a cue was shown 50 ms before the dt and about 100 ms before the saccade (D), or a cue was shown 200 ms before the dt and about 250 ms before the saccade (G). (B,E,H) Normalized sensitivity maps. Averaged normalized sensitivity (d') observed across participants and displayed using a color-coded linear scale going between 0.25 and 0.75 (see Materials and methods). Asterisks indicate significant differences (p < 0.05) in sensitivity found between a particular position of the dt and the average of all the other tested positions. (C,F,I) Averaged normalized d’ obtained at four positions of interest (black squares) and at their corresponding surrounding positions (dark gray squares). Error bars show SEM and asterisks indicate significant comparisons (p < 0.05).

https://doi.org/10.7554/eLife.37598.003

Next, we analyzed the trials during which we presented an additional cue during saccade preparation, with the cue and discrimination target shown shortly after each other (50 ms) and on average 96.88 ± 0.96 ms (cue offset relative to saccade onset) before the saccade (Figure 2D–F). Here, in addition to the fixation and saccade targets, we were also interested in two further locations: the cue location and the retinal location the cue will occupy after the saccade, that is, the remapped location. Visual sensitivity was higher at the cue (0.78 ± 0.07, p = 0.0002), at the saccade target (0.74 ± 0.07, p < 0.0001) and at the fixation target (0.55 ± 0.05, p = 0.0060), when compared to the average of all the tested locations (0.44 ± 0.03). When compared to their closest surrounding positions (Figure 2F), we found spatially specific effects only at the cue (surround: 0.37 ± 0.03, p < 0.0001) and at the saccade target (surround: 0.40 ± 0.03, p < 0.0001), but not at the fixation target (surround: 0.49 ± 0.03, = 0.1112). Importantly, when the cue was shown shortly before the saccade, the visual sensitivity at its remapped location (0.46 ± 0.07) was not significantly higher relative to the other tested positions (0.44 ± 0.03, p = 0.6886). Thus, we observed no evidence for attentional remapping of the cued location when the cue appeared shortly before saccade onset.

These results contrasted with those found in trials when the same cue was shown substantially before the discrimination target (200 ms) and on average 240.82 ± 1.42 ms before the saccade onset (Figure 2G–I). Under such conditions, when compared with the average across all positions (0.45 ± 0.08), we found higher visual sensitivity at the saccade target (0.78 ± 0.08, p = 0.0004), at the cue (0.71 ± 0.06, p < 0.0001) and, critically, at the remapped location of the cue (0.56 ± 0.07, p = 0.0072). Similar to other experimental conditions, the benefits observed at the saccade target (surround: 0.46 ± 0.03, p = 0.0010), at the cue (surround: 0.41 ± 0.04, p < 0.0001) and at its remapped location (surround: 0.45 ± 0.04, p = 0.0382) did not spread towards their respective adjacent positions.

Finally, we found that the increase in sensitivity observed at the remapped location of the cue was present for the condition in which the cue appeared substantially before the discrimination target and the saccade onset, but not if it appeared later. Such an effect was evident from comparison of normalized sensitivity obtained at the remapped position of the cue in the condition in which no cue was shown (0.38 ± 0.05) to conditions in which the cue appeared substantially before the discrimination target and the saccade onset (0.56 ± 0.07, p < 0.0038) or to conditions in which it appeared later (0.46 ± 0.07, p = 0.2816). These comparisons can be visualized by mapping subtraction of the normalized sensitivity obtained in the conditions in which we displayed a cue from those in which no cue was shown (see Figure 3 and Materials and methods). We normalized these differences, to present data with the same color scale as in the condition maps (Figure 2B–2E–2H). Combined, our results so far show that when the visual system is given enough time to process and attend a visual stimulus, such as the salient cue used in our task, spatial attention is remapped to the retinal location the stimulus will occupy after a saccade.

Cue vs. no-cue subtraction maps.

Individual normalized sensitivity (d') is subtracted between conditions and the difference is normalized to obtain maps with convention as in Figure 2. Subtractions are made between trials in which the cue was shown ~100 ms before the saccade (A) or ~250 ms before the saccade (B) to trials in which it was not shown.

https://doi.org/10.7554/eLife.37598.004

If attention is remapped pre-saccadically, one should also expect to find spatially specific attentional effects at the fixation target, as the fixation target is the remapped location of the saccade target. Indeed, one study reported such foveal remapping of the saccade target (Rolfs et al., 2011). In the experiment above, we observed inconsistent evidence for spatial attention at the fixation target (see Figure 2C, F and I). This is likely to be because of the threshold procedure differences between this study and the one that reported spatial remapping of attention to fixation (Rolfs et al., 2011). Indeed, as we were principally interested in remapping of the salient peripheral cue in the above study, our threshold procedure measured spatial attention during fixation (yielding, importantly, no threshold difference between the cue and the remapped location of the cue). However, previous research has shown that preparation of saccades draws spatial attention away from other, non-saccade or non-salient locations (Jonikaitis and Theeuwes, 2013), resulting in an increase of acuity threshold in between the fixation and the saccade target locations (Campbell and Wurtz, 1978). In the experiment described above, we measured perceptual thresholds in a fixation task, rather than in a saccade task, so it is possible that the threshold for the fixation location underestimated that during saccadic preparation. If this was the case, potential remapping of the saccade location to the fixation location could have been masked by drawing attention away from the fixation location in the threshold task. Therefore, in the threshold task of a second experiment (foveal remapping task), we adjusted the tilt of the discrimination target using a saccade rather than a fixation task (see foveal remapping main task in Materials and methods). We observed that the discrimination target had to be tilted by 15.03 ± 2.34° if presented at the fixation target before a saccade (Figure 1D), a tilt up to four times bigger than that recorded during the fixation threshold procedure (Figure 1C). We used these values in a simplified version of the above experiment, without presentation of the cue, and as before, with a discrimination target randomly shown across trials at 32 possible positions while participants prepared a visually guided saccade. If anything, this procedure should be even more sensitive at detecting attention spread between saccade and fixation targets (Zirnsak and Moore, 2014).

Then in this second task, in which only the thresholding procedure was changed, we first analyzed whether the discrimination target affected saccade latency and accuracy. We found a main effect of the eccentricity of the discrimination target on saccade latency within trials in which the visual streams overlapped with the fixation and the saccade targets (F3,21 = 8.89, p = 0.0005, ηp2= 4.84 %), but not on trials when the visual streams did not overlap with the targets (F3,21 = 3.01, p = 0.0532, ηp2= 2.03 %). Further, discrimination target eccentricity did not affect saccade accuracy (F4,28 = 1.60, p = 0.2007, ηp2= 2.53 %). These results suggest that, similar to the first experiment, presentation of the discrimination target had limited influence on preparation of the saccade. Next, and in agreement with the results of the first experiment, we found a systematic pre-saccadic deployment of attention towards the saccade target (Figure 4A, 0.68 ± 0.11, p < 0.0001) when compared with the average over all the tested positions (0.36 ± 0.03). Critically, this benefit was accompanied by systematic deployment of attention at the fixation target (0.96 ± 0.02, p < 0.0001). Finally, these effects were spatially specific (Figure 4B), as shown by the significant differences observed when comparing sensitivity at the fixation target (surround: 0.47 ± 0.06, p < 0.0001) and the saccade target (surround: 0.33 ± 0.04, p < 0.0001) with their relative surrounds.

Discussion

We constructed spatial attention maps by measuring orientation sensitivity while participants prepared a visually guided saccade. Our paradigm allowed us to measure whether attention spreads broadly around the saccade target or shifts towards spatially specific loci. We observed that attention consistently shifted to the saccade target location and, importantly, did not spread to other locations surrounding it. In our main manipulation, we presented a salient cue during saccade preparation. On these trials, we observed a second spatially specific locus of attention at the cued location. Importantly, on these cued trials, we also observed a third, distinct locus of attention. Although this third location was neither salient, nor task-relevant before the saccade, it corresponded to the retinotopic location the cue would occupy after the saccade. In other words, we observed attentional remapping of the cue location before the saccade onset. Critically, these effects were present only when the cue appeared long enough before the saccade onset. This indicates that remapping, like any other attentional process, requires some time. As we observed three separate foci of attention (at the saccade target, cue and remapped location), we did not find support for the hypothesis that attention spreads during saccade preparation around the saccade target (Zirnsak and Moore, 2014).

Our findings speak to the current debate on whether forward remapping exists and what role it plays in maintaining visual stability. As we report behavioral results, it is important to note that the link between our effects and neurophysiology is only theoretical. Behavioral experiments do not allow for direct conclusions as to which neural structures are involved in remapping nor on the validity of neurophysiology findings. However, our findings are relevant to the behavioural predictions provided in earlier neurophysiology work. Combined behavioral and neuronal recordings are necessary to eventually link proposals of neuronal activity before saccades and behavioral observations reported here and previously.

Behavioral studies on visual remapping of attention (Jonikaitis et al., 2013; Rolfs et al., 2011; Szinte et al., 2015; Szinte et al., 2016; Yao et al., 2016b) have been inspired by earlier neurophysiology work suggesting that receptive fields in FEF, LIP and SC shift (or are remapped) in anticipation of post-saccadic stimulus location (Duhamel et al., 1992; Sommer and Wurtz, 2006; Walker et al., 1995). Such a predictive receptive field shift occurs either shortly after a saccade (at a neural latency too short for visual responses) or even before a saccade onset (Colby et al., 1996; Kusunoki and Goldberg, 2003). This so-called forward remapping offers an excellent candidate mechanism of space constancy that has been incorporated into such phenomenon models. In these models, connections between visual neurons with receptive fields spatially separated by the saccade vector can be used to predict post-saccadic visual stimulus location and compensate for retinal image shifts during the saccade (Neupane et al., 2017; Quaia et al., 1998; Wang et al., 2016). However, forward remapping models have been challenged by Zirnsak and colleagues (Chen et al., 2018; Zirnsak et al., 2014), whose results suggested that the early findings were affected by the low spatial resolution of the receptive field mapping technique (Duhamel et al., 1992; Sommer and Wurtz, 2006; Walker et al., 1995). More detailed FEF receptive field mapping suggested that, before a saccade, cells preferentially respond to stimuli presented near the saccade target rather than at the remapped target location. In other words, cell receptive fields do not shift in parallel to the saccade vector, but instead converge towards the saccade target. Zirnsak and colleagues thus argued that forward remapping models cannot explain space constancy and instead one should focus on saccade target selection as a mechanism mediating this phenomenon (Zirnsak and Moore, 2014). More recent work has indicated that the visual system may, in fact, implement both forward and convergent remapping of receptive fields in area V4 (Neupane et al., 2016b; Neupane et al., 2016a). As this combined approach has been criticized on technical grounds (Hartmann et al., 2017), the neurophysiological results regarding the existence of forward remapping remain inconclusive.

On the other hand, a number of behavioral studies have repeatedly and reliably demonstrated forward remapping of spatial attention before saccades (Jonikaitis et al., 2013; Rolfs et al., 2011; Szinte et al., 2015; Szinte et al., 2016; Yao et al., 2016b). The behavioral studies, however, suffered from the same drawbacks as early neurophysiological studies, low spatial resolution. Our study has eliminated this potential criticism, and our observed deployment of attention in the opposite direction of the saccade and of the cue speaks in favor of the forward remapping effects. Consistent with previous behavioral studies (Baldauf and Deubel, 2008; Deubel and Schneider, 1996; Jonikaitis et al., 2017) and contrary to the convergent remapping effects, our results show that spatial attention is allocated to the saccade target and does not broadly spread around it. Additionally, convergent remapping cannot account for a number of earlier behavioral findings (Jonikaitis et al., 2013; Rolfs et al., 2011; Szinte et al., 2015; Szinte et al., 2016), as such spread of attention would have to be asymmetric and not spread towards the several control positions tested in these earlier studies.

Of note, the behavioral consequences of pre-saccadic changes in the spatial tuning of visual cells RF are unclear. Perhaps counterintuitively, recent computational neuroimaging modeling (Kay et al., 2015) has shown that increasing the neural spatial sampling at a particular position, similar to over-sampling of the saccade target observed within convergent remapping studies (Hartmann et al., 2017; Tolias et al., 2001; Zirnsak et al., 2014), results in reduction of spatial uncertainty. Convergent remapping does not necessarily yield a large spread of attention around the saccade goal (Zirnsak and Moore, 2014). Rather, it may increase visual sensitivity to stimuli only in the immediate vicinity of the saccade target. Convergent remapping could, therefore, reflect the spatially specific attentional selection of the saccade target observed in the present study. Our results indicate that spatial visual attention mechanisms must be accounted for in future work of remapping to advance our understanding of space constancy. Here, we hypothesize that previous reports of convergent remapping may reflect increased visual sensitivity at the saccade target explained by both spatial and temporal properties of visual attention.

First, to determine a visual neuron RF spatial profile, neurophysiologists used localized visual probes, presented most of the time in a sparse display with high probe-background contrast. It is, therefore, likely that such probes capture spatial attention (Theeuwes, 1991). The same holds for visual stimuli used to trigger the saccade, which were both task-relevant and, in most experiments, high-contrast stimuli. As visual RFs can shift towards attended locations even without any saccade involved (Niebergall et al., 2011; Womelsdorf et al., 2006), one must account for the effect of attention before interpreting any RF change in spatial tuning. In our study, the attention-capturing cue and the saccade target were dissociated from the measure of spatial attention. We measured attention by a discrimination target that did not significantly capture attention on its own, and, therefore, did not interfere with saccade preparation or pre-saccadic distribution of spatial attention (Deubel and Schneider, 1996). We also use different conditions to account separately for the effect of the saccade target and of the cue. To understand pre-saccadic RF changes in spatial tuning, one should first consider the spatial deployment of spatial attention.

Second, we argue that the temporal dynamics of attention have to be accounted for. We found that the time at which we presented our cue was critical for remapping of attention to be observed before the saccade. In particular, benefits at the remapped location of the cue were observed only when the cue was shown more than 175 ms before the saccade onset. As even the fastest deployment of attention would take a minimum of 100 ms to occur (Ling and Carrasco, 2006; Müller and Rabbitt, 1989; Nakayama and Mackeben, 1989; Rolfs and Carrasco, 2012), our remapping effects are compatible with the time course of attentional selection. Different neurophysiology studies measured visual cell activity followingpresentation of visual objects at different times across saccades (Kusunoki and Goldberg, 2003; Marino and Mazer, 2018; Nakamura and Colby, 2002; Wang et al., 2016). In particular, it was reported that a fair proportion of visual neurons recorded within LIP (Kusunoki and Goldberg, 2003) and earlier visual areas (Nakamura and Colby, 2002) showed forward remapping activity for probes presented as early as 250 ms before the saccade. Interestingly, they observed forward remapping activity preceding the saccade onset only if a visual object was flashed early before the saccade, such that visual objects shown earlier relative to the eye movement resulted in delayed activity during or after the saccade onset (Kusunoki and Goldberg, 2003; Nakamura and Colby, 2002). Our results are consistent with these findings, but not with those of Wang and colleagues (Wang et al., 2016), which report that the presentation of a visual object before the saccade resulted only in post-saccadic forward remapping activities within a set of recorded cells in LIP. Moreover, they found that the recorded cells responded transiently to the intermediate position between the current and the future receptive field position. By contrast, we did not observe any benefit at locations between the cue and the remapped position of the cue. These effects suggest that remapping of attention better reflects the activity of early visual areas than attentional priority maps (LIP, FEF, SC).

It is important to note that neurophysiology studies that failed to observe pre-saccadic forward remapping typically presented probes shortly before (50 ms) the saccade. Our behavioral findings suggest that a short time window between probe presentation and neural recording, which some studies have used, might be insufficient for probes to be remapped to a location parallel to the saccade before the onset of the movement. Further, if remapping is closely related to the time course of attention, it is possible that for attended stimuli shown just prior to the saccade onset, remapping may occur during or after the saccade (Kusunoki and Goldberg, 2003; Nakamura and Colby, 2002). Indeed, Neupane and colleagues (Neupane et al., 2016a; Neupane et al., 2016b) observed forward remapping when measuring post-saccadic memory responses to probes shown just before the saccade. Also, Yao and colleagues (Yao et al., 2016a) have shown that a post-saccadic memory trace of remapping was enhanced by attentional modulations established before the saccade, corroborating the notion that forward remapping can occur after the saccade. Although we did not measure whether a cue shown shortly before saccade onset was remapped after the saccade in the present study, this was done in two previous studies (Jonikaitis et al., 2013; Yao et al., 2016b). Both studies found that spatial attention was allocated to the location of a salient stimulus immediately after the saccade, even when the stimulus was no longer present (Jonikaitis et al., 2013). This indicates that the visual system anticipates the attended stimulus location after a saccade and recomputes its retinotopic location before the saccade is done. Finally, we also observed high perceptual performance at fixation, a result in line with two previous studies that investigated foveal remapping effects (Knapen et al., 2016; Rolfs et al., 2011). Such an attentional effect is surprising, as one would expect that visual selection should prioritize the saccade target, whereas the current fixation should be the least informative and least attended part of the display (especially given that participants already fixated for ~1 s before starting saccade preparation). However, if one considers that fixation-centered receptive fields will process the saccade target after the saccade, forward remapping effects suggest significant attentional benefits at that location, as we observed here and in a previous study (Rolfs et al., 2011).

Marino and Mazer (Marino and Mazer, 2018) recently showed that attention modulates the state of neurons in V4 before saccade onset, without a substantial shift of neurons’ spatial tuning. In contrast to our study, which manipulated transient attention by cueing a location on every trial, they used a sustained attention task to a cue presented before a series of records. Their pre-saccadic effects are consistent with the results reported here and suggest that new models of space constancy should account for the dynamics of spatial attention.

In summary, we used an eccentricity-adjusted discrimination task to measure, for the first time, spatial attention maps before saccades. Using this method, we observed a spatially specific increase in visual sensitivity at the fixation target, the saccade target, the cue and the remapped location of the cue. We found no evidence supporting the convergent remapping interpretation, which suggests that spatial attention spreads around the saccade target in a spatially unspecific way. We found that, before a saccade, attention is deployed towards the saccade target as well as towards a cued location. Further, given that the cue was presented sufficiently early before the saccade, we observed a deployment of attention to its remapped location, that is parallel and opposite to the saccade vector. Although the benefit at that location is smaller compared with that at the cue location, it reflects an ongoing process that facilitates spatial attention allocation after the saccade despite the retinotopic shifts induced by the eye movement.

Materials and methods

Participants

Eighteen students (14 participants in the peripheral remapping task, eight participants in the foveal remapping task, four participants did both tasks) of the Ludwig-Maximilians-Universitä München participated in the experiment (ages 22–30, 10 female, one author), for a compensation of 10 Euros per hour of testing. All participants except the author were naive as to the purpose of the study and all had normal or corrected-to-normal vision. The experiments were undertaken with the understanding and written informed consent of all participants and were carried out in accordance with the Declaration of Helsinki. Experiments were designed according to the ethical requirements specified by the Faculty for Psychology and Pedagogics of the LMU München (approval number 13_b_2015) for experiments involving eye tracking. All participants provided written informed consent, including a consent to publish anonymized data.

Setup

Participants sat in a quiet and dimly illuminated room, with their head positioned on a chin and forehead rest. The experiment was controlled by an Apple iMac Intel Core i5 computer (Cupertino, CA, USA). Manual responses were recorded via a standard keyboard. The dominant eye’s gaze position was recorded and available online using an EyeLink 1000 Desktop Mounted (SR Research, Osgoode, Ontario, Canada, RRID:SCR_009602) at a sampling rate of 1 kHz. The experimental software controlling the display, the response collection as well as the eye tracking was implemented in Matlab (MathWorks, Natick, MA, USA, RRID:SCR_001622), using the Psychophysics (Brainard, 1997; Pelli, 1997) and EyeLink toolboxes (Cornelissen et al., 2002). Stimuli were presented at a viewing distance of 60 cm, on a 21-in gamma-linearized Sony GDM-F500R CRT screen (Tokyo, Japan) with a spatial resolution of 1024 × 768 pixels and a vertical refresh rate of 120 Hz.

Procedure

We completed two different tasks (peripheral remapping and foveal remapping) in a total of four experimental sessions (on different days) of about 100 min each (including breaks). Each task was always preceded by a discrimination threshold measurement, completed at the beginning of each session. Each session was composed of two blocks of the threshold task followed by four to six blocks of the main task. Participants ran a total of 11–12 blocks of the peripheral remapping task and four blocks of the foveal remapping task. Participants who completed the two tasks always started with the peripheral remapping task.

Peripheral remapping task

Each trial began with participants fixating a fixation target, a red frame (2.2 dva/side, 10’ width, 30 cd/m2) presented on a gray background (60 cd/m2). When the participant’s gaze was detected within a 2.0 dva radius virtual circle centered on the fixation target for at least 200 ms, the trial began with a random fixation period of 500–900 ms (uniform distribution, in steps of 50 ms). After this period, the fixation target was replaced by the saccade target (same red frame) presented 12 dva to the right or to the left of the fixation target (Figure 1A). Participants were instructed to move their eyes as quickly and as accurately as possible towards the center of the saccade target. From the beginning of the trial, we presented 12 flickering visual streams (40 Hz), composed of 25 ms vertical Gabor patches (frequency: 2.5 cycles per degree; 100% contrast; same random phase on each screen refresh; standard deviation of the Gaussian window: 0.9 dva; mean luminance: 60 cd/m2) alternating with 25 ms pixel noise square masks (2.2 dva side, made of ~0.04 dva-width pixels). The visual streams were arranged in three by four matrix, with a distance of 6 dva between each element (Figure 1B). On each trial the matrix of 12 visual streams was presented at one out of 15 different positions relative to the display center (shifted by −6 dva, −3 dva, 0, +3 dva or +6 dva vertically and −3 dva, 0 or +3 dva horizontally). Between 50 and 175 ms after the saccade target onset, one of the 12 vertical Gabor patches was replaced by a discrimination target, a tilted Gabor (clockwise or counterclockwise from the vertical). The time interval of 50–175 ms was determined in a pilot study with two criteria: i) that discrimination target offset occurred in the last 150 ms before the saccade, and ii) taking into account shorter saccade latency on trials when the fixation and saccade targets were not covered by visual streams. Once the discrimination target had appeared, no more vertical Gabor patches were presented and only noise masks alternated with blank frames (Figure 1B). Across trials the discrimination target was shown at 32 positions covering 24 dva horizontally and 18 dva vertically (position located at every second intersection of a nine column by seven rows grid, see Figure 1C–D). At the end of each trial, participants reported the orientation of the discrimination target using the keyboard (right or left arrow keys), followed by a negative-feedback sound on error trials.

In two-thirds of the trials we captured attention by presenting a task-irrelevant cue, a 50 ms abrupt color onset stimulus presented in between the fixation and saccade targets, 6 dva above or below the screen center. This cue was a green Gaussian patch (mean luminance of 80 cd/m2), with the same Gaussian window of the Gabors and covering one of the visual streams. Across trials this cue was presented either 50 ms or 200 ms before the discrimination target onset. In one-third of the trials the cue was not presented at all. To avoid inter-trial attention-lingering effect at the cue location, we separated cue and no-cue trials, with no-cue trials presented in the first four blocks of the task.

This method allowed us to map the allocation of attention at four positions of interest: the fixation target, the saccade target, the cue, and the remapped position of the cue, as well as 28 control positions. To maximize the number of trials at the different positions of interest, we presented discrimination targets less often (30% less) in the two rows (nine positions) most distant from the cue (e.g. if the cue was presented above the horizontal meridian, the discrimination target was presented less frequently at the two bottom rows below the horizontal meridian). In the trials without a cue, discrimination targets were presented less often (30% less) in the two most peripheral rows (nine top and nine bottom row) maximizing the number trials around the fixation and the saccade target. This procedure was selected mainly to reduce the duration of the experiment. We are confident that reducing the frequency of presenting a discrimination target at these control positions did not affect the deployment of attention. As in previous studies using the same stimulus (Jonikaitis et al., 2013; Rolfs et al., 2011), we observed at these control positions performance near chance level suggesting that participants are more or less unable to detect the occurrence of the discrimination target. Importantly, we reproduced all the effects presented above, but instead of using all the control positions, we used only those matched in the frequency of discrimination target across trials.

Participants completed between 2914 and 3323 trials of the peripheral remapping task. We checked correct fixation maintenance and correct saccade execution online and repeated incorrect trials at the end of each block. We also repeated trials during which a saccade started within the first 25 ms or ended after more than 350 ms following the saccade target onset (participants repeated between 159 and 443 trials). On average, we analyzed 25.38 ± 0.76 trials and 16.13 ± 0.42 trials per participant at the frequently and less frequently tested positions, respectively, in each of the three main conditions.

Peripheral remapping threshold task

On each session, before the peripheral remapping task, participants completed a threshold task. This allowed us to avoid possible effects of task learning across different sessions as well as to adjust the discrimination target tilt for different eccentricities from fixation. This latter point is particularly important as it reduced the impact of visual acuity (Paradiso and Carney, 1988) on the measure of spatial attention. The threshold task was identical to the main task with the exception that participants kept fixation and the saccade target was not shown. After a random initial period of 500–900 ms (uniform distribution, in steps of 50 ms), a cue was briefly (50 ms) shown followed by a discrimination target 200 ms later at the cued location. Across trials the cue was shown at each of the 32 locations used in the main experiment. The positions of discrimination target and cue were subdivided into five equiprobable groups of eccentricity from the fixation target (eccentricity 1: the fixation target; eccentricity 2: from ~4.2 to 6 dva, eccentricity 3: from ~8.5 to ~9.5 dva; eccentricity 4: from 12 to ~13.4 dva; and eccentricity 5: from ~15.3 to ~16.2 dva). We used a procedure of constant stimuli and a randomly selected orientation of the discrimination target from a linearly spaced interval for each eccentricity (each interval divided into five steps; between ±1 and ±9 dva for the first eccentricity, between ±1 and ±13 dva for eccentricity two and between ±1 and ±17 dva for eccentricities three to five).

Participants were instructed that the cue indicated the position of the discrimination target and were told to report at the end of each trial its orientation (clockwise or counterclockwise). They completed 400 trials across two blocks of the threshold task. For each participant and experimental session individually, we determined for the five eccentricities from the fixation target, five threshold values corresponding to the discrimination target tilts leading to a correct discrimination in 85% of the trials. To do so, we fitted five cumulative Gaussian functions to performance gathered in the threshold blocks. These tilts were used in the main task at their respective eccentricity from the fixation target. In the main task, only trials in which the discrimination target was presented within 150 ms before the saccade were analyzed. As during this period participants are preparing the saccade, any change of orientation sensitivity over space is attributed to the saccadic planning and/or to localized deployment of attention.

Foveal remapping and threshold tasks

The foveal remapping task was identical to the peripheral remapping task, with the exception that we did not present the cue. Moreover, the foveal remapping threshold task differed from the peripheral remapping threshold task, as participants made a saccade during the threshold task instead of keeping fixation (Rolfs et al., 2011). In the foveal remapping threshold task the saccade target could be presented at any of the 32 locations tested. The discrimination target appeared 200 ms after the appearance of the saccade target and participants were instructed that the discrimination target could appear at either fixation or saccade target. We again used a procedure of constant stimuli, and chose discrimination target orientation randomly for intervals defined for the different eccentricities (intervals divided into five steps for each eccentricity; intervals between ±1 and ±25 dva for the eccentricities one and two and between ±1 and ±21 dva for eccentricities three to five).

Participants completed between 973 and 1137 trials of the foveal remapping main task. We checked correct fixation maintenance and correct saccade execution online and repeated incorrect trials at the end of each block. We also repeated trials during which a saccade started within the first 25 ms or ended after more than 350 ms following saccade target onset (participants repeated between 13 and 177 trials). On average, we analyzed 32.12 ± 0.82 trials and 15.73 ± 0.64 trials per participant at the frequently and less frequently tested positions, respectively. Participants completed 500 trials across two blocks of the threshold task. For each participant and experimental session individually, we determined for the five eccentricities from the fixation target, five threshold values corresponding to the angles leading to correct orientation discrimination in 85% of the trials following the same procedure as in the peripheral threshold task.

Data pre-processing

Recorded eye position data were processed offline (independent of online tracking during the experiment). Saccades were detected based on their velocity distribution (Engbert and Mergenthaler, 2006) using a moving average over 20 subsequent eye position samples. Saccade onset was detected when the velocity exceeded the median of the moving average by 3 SDs for at least 20 ms. We included trials if a correct fixation was maintained within a 2.0 dva radius centered on the fixation target, if a correct saccade started at the fixation target and landed within a 2.0 dva radius centered on the saccade target, and if no blink occurred during the trial. Finally, only trials in which the discrimination target disappeared in the last 150 ms preceding saccades were used in the analysis. In total, we included 36,236 trials (90.41% of the online accepted trials, 82.20% of all trials) in the peripheral remapping main task and 7306 trials (95.13% of the online accepted trials, 86.85% of all trials) of the foveal remapping main task.

Behavioral data analysis

Data were analyzed separately for the three conditions of the peripheral remapping task and the only condition of the foveal remapping task. For the trials in which a cue was presented, the cue onset preceded the discrimination target onset by either 50 or 200 ms. Therefore, one condition included the trials with a SOA of 50 ms during which the cue disappeared in the last 175 ms before the saccade, and a second condition included the trials with a SOA of 200 ms during which the cue disappeared more than 175 ms before the saccade. A third condition of the peripheral remapping task and all the trials of the foveal remapping task included trials in which no cue was shown.

To map the allocation of attention, we first mirrored discrimination target positions of leftward saccade trials to match those of the rightward saccade trials. Moreover, in trials with a cue, we mirrored positions of the bottom cue trials (trials in which the cue was shown 6 dva below the screen center) to match those of the top cue trials. Then, for each participant and each condition, we determined the sensitivity in discriminating the orientation of the discrimination target (d’): d’=z(hit rate) - z(false alarm rate). To do so, we defined a clockwise response to a clockwise discrimination target (arbitrarily) as a hit and a clockwise response to a counterclockwise discrimination target as a false alarm. Corrected performance of 99% and 1% were substituted if the observed proportion correct was equal to 100% or 0%, respectively. Performance values below the chance level (50% or d’ = 0) were transformed to negative d’ values. We next normalized for each participant individually, the sensitivity obtained at each position by the range obtained across all tested positions following this formula d’n = (d’n - min) / (max - min), with d’n the sensitivity at a given n position, min and max, respectively, the minimum and maximum sensitivity obtained across the 32 tested positions in the specific condition. These normalized values were then averaged across participants and used to plot sensitivity maps and to perform statistical comparisons. Subtraction maps (Figure 3) were obtained by first subtracting the normalized sensitivity difference between trials in which the cue was presented 50 ms or 200 ms before the discrimination target onset from trials in which no cue was shown. For this comparison we used normalized sensitivity of raw sensitivity as the conditions differ both in the number of trials per discrimination target and in the number of expected salient locations across conditions. These difference values were then normalized across the position according to the same formula as above and averaged across participants to highlight differences between the two obtained subtraction maps.

We obtained sensitivity maps (Figures 2B, E, H, 3A, B and 4A), by interpolating (triangulation-based natural neighbor interpolation) the missing values located at every second intersection of the nine columns by seven rows grid of discrimination targets using the 32 tested positions. We then rescaled the grid (Lanczos resampling method) to obtain a finer spatial grain. We drew position sensitivity maps across participants as colored maps coding the mean sensitivity across participants following a linear color scale going from 0.25 to 0.75 normalized sensitivity. To determine the threshold maps (Figure 1C–D), we first interpolated (linear interpolation) the mean threshold angle obtained for each participant individually over the five different distances between the fixation target and the discrimination target. Map of threshold dt angles across participants was then obtained by drawing disks centered on the fixation target with a radius corresponding to the eccentricity at which the discrimination target was played and coded the mean threshold angle obtained across participants following a linear color scale going from 0° to 20° of discrimination target tilt.

Foveal remapping task results.

(A) Normalized sensitivity maps. Averaged normalized sensitivity (d'). (B) Averaged normalized sensitivity (d')obtained at two positions of interest (see center in black) and at their corresponding surround positions (see surround in dark gray). Conventions and color scale are as in Figure 2.

https://doi.org/10.7554/eLife.37598.005

For statistical comparisons we drew (with replacement) 10000 bootstrap samples from the original pair of compared values. We then calculated the difference of these bootstrapped samples and derived two-tailed p values from the distribution of these differences. Statistical comparisons of the eccentricity effect of the discrimination target on different saccade metrics (saccade latency and accuracy) were tested using repeated measures ANOVA. Discrimination target positions were grouped depending on their eccentricities from the fixation target as defined in the threshold tasks.

References

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
  42. 42
  43. 43
  44. 44
  45. 45
  46. 46
  47. 47
  48. 48

Decision letter

  1. Andrew J King
    Senior and Reviewing Editor; University of Oxford, United Kingdom

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your article "Pre-saccadic remapping relies on dynamics of spatial attention" for consideration by eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by Sabine Kastner as the Reviewing and Senior Editor. The reviewers have opted to remain anonymous.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

Summary

The study conducted by Szinte et al. attempts to resolve the debate whether the spatial updating mechanism linked to saccade planning and execution that was discovered by neurophysiological studies has a psychophysical counterpart. The experiment was designed on the premise that spatial updating operates on locations in the visual field where attention is deployed, such as the target of an eye movement or an attention-soliciting cue. Using a high-resolution mapping procedure coupled with a visual discrimination task, the authors tried to determine if psychophysical thresholds are reduced (i.e. visual discrimination improves) in three key regions where attentional resources could be allocated: 1. the position of the saccade target, 2. that of an attention cue and, critically, 3. the "remapped" location of that cue, i.e. the retinotopic location it will occupy as a consequence of the saccade. Consistent with previous work, discrimination performances were found to be improved in close vicinity of the saccade target and of the attention cue. Furthermore, the results provide support in favor of the forward remapping hypothesis, by showing a small but significant enhancement of discrimination performance at a location distant from the attention cue but which corresponds precisely to the retinotopic receptive field of the visual neurons that will encode this cue at the end of the saccade. Critically, the effect is observed only if the cue is presented more than 200ms before the saccade, indicating that attentional instruction has to be fully processed for the remapping to take place.

All reviewers agreed that this is a carefully and timely study and makes an important contribution to the ongoing debate on spatial remapping.

Essential revisions:

1) Experimental design concerns. In particular, it was unclear how the eccentricity-based orientation thresholding was applied. While the benefits of the design were appreciated, it is critical that these thresholds be properly selected, otherwise it might affect the pattern of results, because the analysis is directly comparing sensitivity across locations that had different difficulty levels. In order to address this concern, the sensitivity at each location should be compared to the baseline no-cue condition at that location (i.e., subtract Figure 2B from 2E and 2F). This would alleviate concerns about directly comparing across locations that may not have been properly equated. It would also resolve a related concern, which is that in the Materials and methods it is revealed that not all target locations were probed with the same frequency. This suggests that it may be misleading to test sensitivity at a given location relative to the average of all tested locations; rather, it should be compared to the average of all locations probed with the same frequency – or better yet, to that exact location under the no-cue baseline condition.

This essential analysis would also alleviate a related point, that is, the timing of forward remapping does not match with neurophysiology. Firstly, in paragraph three of the Introduction, the authors mention that "contrary to neurophysiology", they manipulated the timing of stimulus onset relative to the saccade. There are, however, neurophysiological studies that explicitly looked at remapping vs timing of probes (Kusunoki and Goldberg, 2003; Nakamura and Colby, 2002; Wang, et al., 2016). The authors should correct this sentence. Secondly, all of these neurophysiology studies show that the strength of forward remapping increases with decreasing probe-to-saccade onset temporal distance. This apparent mismatch between the authors' observation and neurophysiology could be because they didn't account for changing discrimination threshold across pre-saccadic time period either due to changing visual acuity (Campbell and Wurtz, 1978) or changing strength of spatial attention (Figure 1 C,D). Thirdly, it could simply be that forward remapping could have been observed if probes were flashed after the saccade (similar to what was done by Jonikitis et al., 2013; we acknowledge that the authors do mention this in discussion). These different possibilities should be discussed in the context of the results from the control analysis.

2) Some of the conclusions should be toned down, and the framing should be clarified:

i) The authors appear to over-reach in the implications of this study for the neural receptive field remapping debate. The neural debate is whether receptive fields spatially shift in a forward or convergent manner. The current study is a behavioral study of attention. The authors find some very interesting results, but it is unclear whether any definitive conclusions can be drawn about visual receptive field remapping from a behavioral study of attention. E.g., the conclusion at the end of the Abstract is that "pre-saccadic remapping is an attentional process […]". It is not clear how these results show that remapping in general is an attentional process, or that the conclusions about the nature of attentional remapping necessarily generalize to remapping in general. It may be preferable to contextualize these results in the context of "remapping of attention" only, and to include some discussion of studies that have debated whether the process of remapping involves receptive fields shifting spatially vs whether remapping is more of an attentional process (e.g. Cavanagh et al., 2010; Marino and Mazer, 2018).

ii) The authors rightly use multiple probe locations to measure remapping (the novelty of this paper and a much needed experiment in the field). But they might be looking at high spatial resolution at the cost of low temporal resolution (a caveat also in Zirnsak et al., 2014). In this study, the caveat is the choice of distractor target (DT) orientation despite a changing threshold for it across time. They are comparing conditions for probes presented long before and shortly before the saccade using the same DT orientation. This may be the reason that no remapping was observed when probes were flashed near the saccade onset. In other words, why isn't it the case that forward remapping wasn't observed for late probes because the DT discrimination threshold was higher then? Indeed visual acuity is known to change near the saccade onset (Campbell and Wurtz, 1978) and the authors also show that the threshold is modulated by saccade (Figure 1C vs D and foveal remapping results in Figure 3). These caveats need to be discussed.

iii) While the study seems to demonstrate convincing evidence in favor of forward remapping, the reviewers and editors were specifically uncomfortable with the strong tone of this paper on the 'lack of convergent remapping'. We think that this particular conclusion will need revision.

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Pre-saccadic remapping relies on dynamics of spatial attention" for further consideration at eLife. Your revised article has been favorably evaluated by Andrew King (Senior Editor), a Reviewing Editor, and two reviewers.

The manuscript has been improved but there are some remaining issues that need to be addressed before a final decision can be made, as outlined by reviewer 1 below:

Reviewer #1:

While I still think the results make an interesting and important contribution to the field (and I am reassured by the no-cue-baseline subtraction analysis), I found the revision unsatisfactory.

1) The strong anti-convergent remapping tone of the paper has not really been toned down. There are places where it is improved, but other places where it seems like the authors have doubled-down on how it "indisputably rejects" that hypothesis.

2) The sentence in the Abstract that was flagged as being problematic for claiming that "pre-saccadic remapping is an attentional process" has not been changed. Moreover, while the discussion is a bit more careful about linking the behavioral and neural results, the tone is less one of framing the current results in terms of remapping of attention (and acknowledging potential limitations of the behavioral approach), and more one of repeatedly criticizing the neural studies for not taking attentional effects into account. This is a fair criticism that absolutely should be mentioned, but it's not enough to just do that.

3) The new subtraction analysis that was presented as an "essential revision" is hidden in the supplement (why? If anything it makes their results even more salient), and there's not enough detail provided. It's not clear what exactly the comparisons are for the statistics given in paragraph five of the Results section: Are these simply t-tests between the normalized d' at the remapped location for no-cue vs early-cue and no-cue vs late cue? If so, what does it mean that "these results were evident even after subtracting the normalized sensitivity"? Moreover, what about the other stats presented for the main analyses? The supplemental figure only shows normalized sensitivity maps, not the plots of d' at the positions of interest and their surround positions. Finally, the Materials and methods themselves are unclear. In subsection “Behavioral data analysis” it states that subtraction maps were obtained by "subtracting the normalized sensitivity difference". First, I'm assuming they meant subtracting the normalized sensitivity "values"? (I.e., it wasn't actually a subtraction of difference scores?) Second, what is the rationale for the 2-step normalization: subtracting the normalized sensitivities and then doing a second normalization on those difference values? Wouldn't it be better to subtract the raw sensitivity scores (and then only apply the normalization on the difference maps)?

Reviewer #3:

The authors have addressed all the concerns. I am satisfied with the paper and it qualifies for publication. I do have some reservations with their interpretation of Kusunoki and Goldberg, 2003, Nakamura and Colby, 2002, and Wang et al., 2016, but my reservations are beyond the scope of this paper.

https://doi.org/10.7554/eLife.37598.010

Author response

Essential revisions:

1) Experimental design concerns. In particular, it was unclear how the eccentricity-based orientation thresholding was applied. While the benefits of the design were appreciated, it is critical that these thresholds be properly selected, otherwise it might affect the pattern of results, because the analysis is directly comparing sensitivity across locations that had different difficulty levels. In order to address this concern, the sensitivity at each location should be compared to the baseline no-cue condition at that location (i.e., subtract Figure 2B from 2E and 2F). This would alleviate concerns about directly comparing across locations that may not have been properly equated. It would also resolve a related concern, which is that in the Materials and methods it is revealed that not all target locations were probed with the same frequency. This suggests that it may be misleading to test sensitivity at a given location relative to the average of all tested locations; rather, it should be compared to the average of all locations probed with the same frequency – or better yet, to that exact location under the no-cue baseline condition.

We extended and clarified our description of the threshold procedure in the Materials and method and Results sections. We believe our threshold procedure was necessary to evaluate maps of sensitivity without eccentricity-based effects. Indeed, the absence of an eccentricity gradient in the observed sensitivity maps suggests that this procedure was successful (see Figures 2B, 2E, 2H and Figure 3A). However, we agree with the reviewers that these results cannot guarantee that the threshold procedure had no other effect on our results. As suggested by them, we prepared maps of sensitivity difference between the conditions (see Figure 2—figure supplement 1 and paragraph five of the Results and subsection “Behavioral data analysis”). The obtained maps correspond to what one would expect from a visual subtraction of the Figure 2E and 2H from the Figure 2B, with a presaccadic remapping of attention effect observed only when subtracting the early cue (Figure 2H) condition to the no cue (Figure 2B) condition. Next, as suggested by the comments of the reviewers, we directly compared the position of interest to the “baseline” no cue condition and found results directly compatible with our former analyses and conclusions. In particular, we observed a benefit at the remapping position of the cue only when the no cue condition was compared to the early cue but not to the late cue condition. We now report these results in the revised manuscript (paragraph five of the Results).

From the second part of the comment, we understood that reviewers were concerned about the fact that we compared individual positions of interest to the average of all positions, while some positions were less frequently tested. As pointed out by the reviewers, we did not use the same frequency of testing at all positions, principally to spare our participants the extra effort in an already very long experiment. We reduced the frequency of testing only at positions in which we had good reason to believe participants will not discriminate the targets well. Indeed, we showed in our earlier work (e.g. Rolfs et al., 2011; Jonikaitis, et al. 2013; Wollenberg, et al. 2018) that at similar control positions, sensitivity is close to chance level and overall that the presence of a discrimination target has no observed effect on saccade preparation. In the present study, we replicate these effects and now discuss on these aspects in the revised manuscript (see subsection “Peripheral remapping task”). Nevertheless, following the reviewers’ suggestion, we re-analyzed all the effects reported in the submitted manuscript by comparing each position of interest to all locations assessed with the same frequency. We did not observe any change in the statistics both for the peripheral and the foveal remapping tasks. This reanalysis of our data suggests that our results can’t be explained by a difference in discrimination target frequency across the tested positions. We believe that the discrimination target would have to capture attention for frequency of testing different locations to have an effect on attentional deployment. Our analyses of saccadic latency and saccade accuracy and performance strongly suggest that the location of discrimination target did not, in fact, capture attention. With this in mind, and in the light of the novel, control analyses, we found that the potential impact of testing frequency is minimal. For these reasons, we chose to keep the original analyses in the manuscript and we now mention the novel, control analyses (see subsection “Peripheral remapping task”).

This essential analysis would also alleviate a related point, that is, the timing of forward remapping does not match with neurophysiology. Firstly, in paragraph three of the Introduction, the authors mention that "contrary to neurophysiology", they manipulated the timing of stimulus onset relative to the saccade. There are, however, neurophysiological studies that explicitly looked at remapping vs timing of probes (Kusunoki and Goldberg, 2003; Nakamura and Colby, 2002; Wang, Goldberg et al., 2016). The authors should correct this sentence. Secondly, all of these neurophysiology studies show that the strength of forward remapping increases with decreasing probe-to-saccade onset temporal distance. This apparent mismatch between the authors' observation and neurophysiology could be because they didn't account for changing discrimination threshold across pre-saccadic time period either due to changing visual acuity (Campbell and Wurtz, 1978) or changing strength of spatial attention (Figure 1 C,D). Thirdly, it could simply be that forward remapping could have been observed if probes were flashed after the saccade (similar to what was done by Jonikitis et al., 2013; we acknowledge that the authors do mention this in discussion). These different possibilities should be discussed in the context of the results from the control analysis.

We modified the manuscript and added a section discussing the mentioned papers and their outcomes in regard of our results (see Discussion section). However, we believe that there may be a source of conceptual confusion at play when drawing analogies between psychophysical and neurophysiological studies.

First, we used the term "cue" to refer to a peripheral "probe" that may subsequently be remapped. The cue stimulus is analogous to the electrophysiology probes (flash lights, squares, bars, etc.) shown either inside or outside the recorded pre-saccadic cell’s receptive field. To evaluate the deployment of attention, we measured response accuracy to a "discrimination target" presented before the saccade. In electrophysiology, using discrimination targets is not mandatory as neurophysiologists can directly record cell activity. We observed when the cue (probe) was shown long before the saccade (early cue), a pre-saccadic deployment of attention at the remapped position of the cue. This result is in agreement to what Nakamura and Colby, 2002, and Kusunoki and Goldberg, 2003, observed when they presented probes long before the saccade. If the probe was presented at the future receptive field position, they reported an increase in cells’ activity starting before the execution of the saccade. This effect was observed both for neurons recorded within the brain features maps (V3A, V3, V2, V1) and the brain priority maps (LIP).

When the probe was presented immediately prior to the execution of the saccade (corresponding to our late cue condition) the electrophysiological studies found delayed neural responses, starting either during or after the saccade. As the neurons, in the late probe condition, were unresponsive before the saccade, one should not expect benefits at the remapped position before the saccade. Rather, the remapping benefits should be observed after the saccade. We proposed this explanation already in our submitted manuscript. In the revised manuscript we discuss these issues in more details explicitly relating our findings to the electrophysiological studies (see Discussion section).

The results we observed are, in our opinion, in close agreement with the findings of neurophysiology. They also match with a recent study of Marino and Mazer, 2018, who found a transfer of attentional modulation (hand-off) preceding the saccade onset. It is important to note that in their study the allocation of attention is manipulated by cueing a position before a set of recoding trials. The cue of this study could correspond to our early cue condition and their results match with our observed effects. Finally, our results mismatch with the results of Wang and colleagues (2016). These authors report that LIP remapping activity follows the saccade onset for probes presented both long before and just before the saccade. They, moreover, found that the recorded cell’s respond transiently to the intermediate position between the current and the future receptive field position. We do not observe any benefit in between the cue and the remapped position of the cue. These effects suggest that the remapping of attention might better reflect features than priority maps neurophysiology. We discuss these different studies in the revised manuscript.

2) Some of the conclusions should be toned down, and the framing should be clarified:

i) The authors appear to over-reach in the implications of this study for the neural receptive field remapping debate. The neural debate is whether receptive fields spatially shift in a forward or convergent manner. The current study is a behavioral study of attention. The authors find some very interesting results, but it is unclear whether any definitive conclusions can be drawn about visual receptive field remapping from a behavioral study of attention. E.g., the conclusion at the end of the Abstract is that "pre-saccadic remapping is an attentional process […]". It is not clear how these results show that remapping in general is an attentional process, or that the conclusions about the nature of attentional remapping necessarily generalize to remapping in general. It may be preferable to contextualize these results in the context of "remapping of attention" only, and to include some discussion of studies that have debated whether the process of remapping involves receptive fields shifting spatially vs whether remapping is more of an attentional process (e.g. Cavanagh et al., 2010; Marino and Mazer, 2018).

In the revised manuscript, we frame our results in the context of the “remapping of attention”, toning down different sentences and mainly discussing the receptive field remapping in the Discussion section. In particular we include a discussion of the study of Marino and Mazer (2018, published after the initial submission) who proposed that remapping of attention may operate without a shift of visual receptive fields.

ii) The authors rightly use multiple probe locations to measure remapping (the novelty of this paper and a much needed experiment in the field). But they might be looking at high spatial resolution at the cost of low temporal resolution (a caveat also in Zirnsak et al., 2014). In this study, the caveat is the choice of distractor target (DT) orientation despite a changing threshold for it across time. They are comparing conditions for probes presented long before and shortly before the saccade using the same DT orientation. This may be the reason that no remapping was observed when probes were flashed near the saccade onset. In other words, why isn't it the case that forward remapping wasn't observed for late probes because the DT discrimination threshold was higher then? Indeed visual acuity is known to change near the saccade onset (Campbell and Wurtz, 1978) and the authors also show that the threshold is modulated by saccade (Figure 1C vs D and foveal remapping results in Figure 3). These caveats need to be discussed.

As the reviewers, we also believe that a measure of sensitivity with high spatial resolution was needed for the field and acknowledge that it comes at the cost of a temporal resolution. We opted for a high-spatial resolution to obtain sensitivity maps for discrimination target presented in the last 150 ms before saccade. This allowed us to compare behavior effects with the controversial records of Zirnsak et al., 2016, in which a similar analysis was used. We also opted for a high spatial resolution, as we already reported in previous studies measures with a fine temporal resolution extending both before (Rolfs et al., 2011, Szinte et al., 2015, Szinte et al., 2016) and after the saccade onset (Jonikaitis et al., 2013). Indeed, combining in the same study a fine spatial and temporal resolution of sensitivity would have involved the collection of at least three to five times the number of trials we had, giving about 15h to 25h of testing per participants.

Next, Campbell and Wurtz, 1978, measured acuity threshold across saccades using a Snellen test chart presented in between a fixation and a saccade target. They reported an elevation of the minimal visible angle at these positions in the last 150 ms before the saccade. We observed the same elevation of the threshold at these positions (see Figure 1C and 1D) and mention this study in the revised manuscript. This finding motivated our foveal remapping task in which we focus on the remapping in between the saccade and the fixation target. Importantly, as we used in the peripheral remapping task the same threshold for each of our conditions, and used for both, no cue, early cue and late cue conditions measures obtained with discrimination targets presented at the same time relative to the saccade onset (last 150 ms), any spatially non-specific change of threshold should have impacted equally our pattern of results. In other words, if as suggested by the reviewers and by Campbell and Wurtz, 1978, the acuity threshold would overall increase before the saccade, this effect can’t explain the difference obtained between the conditions, and especially can’t explain the observed lowering of orientation threshold at the remapped position of the cue. We now specify these aspects in the revised manuscript.

iii) While the study seems to demonstrate convincing evidence in favor of forward remapping, the reviewers and editors were specifically uncomfortable with the strong tone of this paper on the 'lack of convergent remapping'. We think that this particular conclusion will need revision.

We toned down our interpretations on the “lack of convergent remapping”, that we now suggest as one alternative explanation of the observed behavioral findings.

[Editors' note: further revisions were requested prior to acceptance, as described below.]

Reviewer #1:

While I still think the results make an interesting and important contribution to the field (and I am reassured by the no-cue-baseline subtraction analysis), I found the revision unsatisfactory.

1) The strong anti-convergent remapping tone of the paper has not really been toned down. There are places where it is improved, but other places where it seems like the authors have doubled-down on how it "indisputably rejects" that hypothesis.

We changed our manuscript to temper our conclusion.

2) The sentence in the Abstract that was flagged as being problematic for claiming that "pre-saccadic remapping is an attentional process" has not been changed. Moreover, while the discussion is a bit more careful about linking the behavioral and neural results, the tone is less one of framing the current results in terms of remapping of attention (and acknowledging potential limitations of the behavioral approach), and more one of repeatedly criticizing the neural studies for not taking attentional effects into account. This is a fair criticism that absolutely should be mentioned, but it's not enough to just do that.

We modified the Abstract and edited our manuscript to acknowledge potential limitations of the behavioral approach (see Discussion paragraph two). We make it very clear that our behavioral approach does not impact interpretation of earlier neurophysiology work. However, as earlier neurophysiology work explicitly proposed effects of convergent remapping on attention, we discuss in detail attentional effects and compare our results with other studies.

3) The new subtraction analysis that was presented as an "essential revision" is hidden in the supplement (why? If anything it makes their results even more salient), and there's not enough detail provided. It's not clear what exactly the comparisons are for the statistics given in paragraph five of the Results section: Are these simply t-tests between the normalized d' at the remapped location for no-cue vs early-cue and no-cue vs late cue? If so, what does it mean that "these results were evident even after subtracting the normalized sensitivity"? Moreover, what about the other stats presented for the main analyses? The supplemental figure only shows normalized sensitivity maps, not the plots of d' at the positions of interest and their surround positions. Finally, the Materials and methods themselves are unclear. In subsection “Behavioral data analysis” it states that subtraction maps were obtained by "subtracting the normalized sensitivity difference". First, I'm assuming they meant subtracting the normalized sensitivity "values"? (I.e., it wasn't actually a subtraction of difference scores?) Second, what is the rationale for the 2-step normalization: subtracting the normalized sensitivities and then doing a second normalization on those difference values? Wouldn't it be better to subtract the raw sensitivity scores (and then only apply the normalization on the difference maps)?

We acknowledge that the revised manuscript was somewhat short on details of this condition comparison analyses and therefore understand Reviewer 1’s concerns. We now provide more details (see Results paragraph six and subsection “Behavioral data analysis”) and include the supplementary figure as a main figure (Figure S1 becomes Figure 3).

First, we would like to clarify the statistical comparisons used. These comparisons, as well as all comparisons in the manuscript, with the exceptions of ANOVA, are obtained by drawing with replacement 10,000 bootstrap samples from the original pair of compared values and later calculate the difference of these bootstrapped samples to derive two-tailed p values from the distribution of these differences. This procedure is described in the Materials and method section.

We deleted the sentence “these results were evident even after subtracting the normalized sensitivity”, which wasn’t clear.

The main comparison that, we believe, speaks to the “essential revision” of Reviewer 1 is described in paragraph six of the Results of the revised manuscript. Briefly, we compared normalized sensitivity obtained at the remapped position of the cue when the cue was presented, with normalized sensitivity obtained at the same position when no cue was shown. We did that analysis and the subtraction figures using normalized values as we understood from the previous review that it was the reviewers’ request (“subtract Figure 2B to Figure 2E and Figure 2F”, which are figures of normalized effects). Importantly, this analysis gives the same effects when instead of comparing normalized d’, one uses comparison of the raw d’ values. In particular we found that sensitivity obtained at the remapped position of the cue in the condition in which no cue was shown (0.01 ± 0.07) significantly differs to the raw sensitivity observed at the same position when the cue appeared substantially before the discrimination target and the saccade onset (0.31 ± 0.09, p < 0.0066) but not if it appeared later (0.12 ± 0.14, p = 0.4828).

Please, note that we do not present in Figure 3 the comparison of center and surround positions, and no longer present the statistics of single position comparison to all other positions as such comparisons go beyond the scope of the analysis that this figure visualizes.

To be consistent with the use of normalized sensitivity in Figure 2 and the main analysis, we would like to keep Figure 3 with the subtraction of the normalized effects. These normalization steps, and the second step, the normalization of the difference, allows to use the same color scale for all figures (Figure 3A vs. Figure 3B) and to visually compare these subtractions maps to the main effects (Figure 3A-3B vs Figure 2B2E-2H).

https://doi.org/10.7554/eLife.37598.011

Article and author information

Author details

  1. Martin Szinte

    Department of Cognitive Psychology, Vrije Universiteit, Amsterdam, The Netherlands
    Contribution
    Conceptualization, Resources, Data curation, Software, Formal analysis, Funding acquisition, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing
    For correspondence
    martin.szinte@gmail.com
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2040-4005
  2. Donatas Jonikaitis

    Department of Neurobiology, Howard Hughes Medical Institute, Stanford University School of Medicine, Stanford, United States
    Contribution
    Conceptualization, Investigation, Methodology, Writing—original draft, Writing—review and editing
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0001-9851-0903
  3. Dragan Rangelov

    Queensland Brain Institute, The University of Queensland, Brisbane, Australia
    Contribution
    Conceptualization, Resources, Investigation, Methodology, Writing—review and editing
    Competing interests
    No competing interests declared
  4. Heiner Deubel

    Allgemeine und Experimentelle Psychologie, Ludwig-Maximilians-Universität München, Munich, Germany
    Contribution
    Conceptualization, Supervision, Methodology, Project administration, Writing—review and editing
    Competing interests
    No competing interests declared

Funding

Deutsche Forschungsgemeinschaft (SZ343/1)

  • Martin Szinte

Deutsche Forschungsgemeinschaft (RA2191/1-1)

  • Dragan Rangelov

H2020 Marie Skłodowska-Curie Actions (704537)

  • Martin Szinte

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

This research was supported by a Deutsche Forschungsgemeinschaft temporary position for principal investigator grant to MS (SZ343/1) and DR (RA2191/1-1), a Marie Sklodowska-Curie Action Individual Fellowship to MS (704537). We are grateful to the members of the Deubel laboratory in Munich for helpful comments and discussions and to Elodie Parison, Alice and Clémence Szinte for their invaluable support.

Ethics

Human subjects: Experiments were designed according to the ethical requirements specified by the Faculty for Psychology and Pedagogics of the Ludwig-Maximilians-Universität München (approval number 13_b_2015) for experiments involving eye tracking. All participants provided written informed consent, including a consent to publish anonymized data.

Senior and Reviewing Editor

  1. Andrew J King, University of Oxford, United Kingdom

Publication history

  1. Received: April 16, 2018
  2. Accepted: December 30, 2018
  3. Accepted Manuscript published: December 31, 2018 (version 1)
  4. Version of Record published: January 10, 2019 (version 2)

Copyright

© 2018, Szinte et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.

Metrics

  • 580
    Page views
  • 101
    Downloads
  • 0
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)

  1. Further reading

Further reading

    1. Neuroscience
    Jessica Ausborn et al.
    Research Advance
    1. Neuroscience
    Kazunori Shinomiya et al.
    Research Article Updated