1 Main

Humans make fast, ballistic eye movements, called saccades, to explore the rich visual world [1]. Saccades are executed approximately three to four times per second [2, 3]. Where to saccade is therefore one of the most frequent decisions the brain is faced with [4].

It is well established that the physical properties of the environment (bottom-up information) [5–9], the goals of the observer (top-down information) [10–13], and prior knowledge about a scene (selection history) [14, 15] drive where the eyes are moved. However, even when these factors are kept constant, there are many systematic biases in eye-movement behavior, such as a bias for cardinal compared with oblique saccade directions [16–21]. The presence of these biases suggests that additional factors must contribute to the decision of where to saccade, here referred to as ‘saccade selection’. Recent evidence suggests that the effort involved with planning and executing (eye) movements may be one crucial factor [22–25]. Effort is thought to be minimized whenever possible [26, 27], likely because it is costly to spend inherently limited cognitive resources [22, 28]. We here use the term ‘saccade cost’ to describe the effort associated with planning and executing saccades. Although saccades are relatively affordable [i.e. not very costly; 1, 29], they are executed very often [2, 3] and therefore even small costs should add up over time [22, 30]. We here hypothesized that affordable saccades are preferred over costly saccades. However, quantifying subtle saccade costs (neuro-)physiologically, and thus directly testing this proposition, proved difficult until recently.

We recently demonstrated that the effort of saccade planning can be measured with pupil size, which allows us to quantify saccade costs as long as low-level visual factors are controlled for [30]. Pupil size is an established marker of activity in the locus coeruleus [31–33], which projects noradrenaline throughout the brain [31, 34–38]. Noradrenaline is thought to modulate neural gain [39, 40], and to coordinate the communication within and between neural populations [41–43]. By capturing these processes, pupil size not only indexes mental effort, for instance when loading working memory [33, 44–54], but also the effort of planning (and executing) movements [55–57]. We leveraged this to demonstrate that saccade costs are captured by pupil size, and are higher for oblique compared with cardinal directions [30]. Here, we addressed whether saccade costs predict where to saccade.

We hypothesized that participants would prefer affordable over costly saccades to minimize effort expenditure. To test this, we first mapped out saccade costs across directions by measuring pupil size during saccade planning. To assess saccade preferences across the same directions, a subsequent free choice saccade task was employed. Previewing our results, saccade costs indeed predicted saccade preferences, as affordable directions were preferred over costly alternatives. Strikingly, this general principle even held when participants searched for targets in natural scenes in two additional experiments: saccade cost remained a fundamental driver of saccade selection. If saccades and other cognitive operations consume the same resources, this should reflect in adaptively changing saccade preferences in light of altering cognitive demands. We tested this idea experimentally by comparing saccade preferences with and without an auditory dual-task. As hypothesized, participants made fewer saccades overall under increased cognitive demand and especially cut the most costly directions. This provides convergent evidence that saccades are costly and rely at least in part on the same cognitive resources as other cognitively demanding operations.

2 Results

2.1 Saccade costs differ around the visual field

Twenty human participants planned and executed saccades in 36 different directions at a fixed amplitude (10°, Figure 1a, b). Pupil size was measured to index effort and thereby saccade cost during saccade planning (−150 ms until 170 ms around cue offset; also see Supplementary Figure 1). Replicating our previous findings, we found that pupil size differed across directions [30] (Figure 1c-f). We observed a larger pupil size during planning of oblique saccades compared with cardinal saccades (β = 7.662, SE = 1.957, t = 3.916, p < .001). Downward saccades were associated with a larger pupil size than upward saccades (β = .556, SE = .171, t = 3.261, p = .001), and a slightly larger pupil size for leftward compared with rightward saccades (β = .226, SE = .095, t = 2.388, p = .017). These effects were not mediated by differences in saccade properties, such as duration, amplitude, peak velocity, and landing precision (Figure 1e, f). Together, this shows that saccade costs differ as a function of direction, indicating that certain saccades are more costly than others.

Figure 1:

2.2 Saccade costs predict saccade preferences

The same twenty participants subsequently completed a saccade preference task adapted from [23]. To determine which of the 36 saccade directions were preferred, participants freely chose between two possible saccade targets every trial (Figure 2a). We first analyzed whether saccade preferences differed across directions. Results showed that participants preferred cardinal over oblique directions (β = .091, SE = .023, t = 3.910, p < .001; Figure 2b), and preferred upward over downward directions (β = .036, SE = .017, t = 2.130, p = .033). No differences were observed between leftward and rightward saccade directions (β = .009, SE = .014, t = .668, p = .504).

Figure 2:

These results indicate that saccade preferences seem to mirror the pattern of saccade costs (compare Figure 1d and Figure 2b). We proceeded to directly test if saccade costs predicted saccade preferences. To this end, we calculated the overall proportion of selections for each direction to index saccade preferences (Figure 2b). In line with our hypothesis, pupil size during saccade planning (in cued directions) negatively correlated with saccade preferences (during self-selection) (r(35) = −.76, p < .001; Figure 2c). For the first time, this demonstrates that saccade costs critically predict saccade preferences. Put differently, participants preferred saccading towards affordable over costly options. Illustrating the robustness of this relationship, we found smaller pupil sizes during saccade planning for preferred (selected >50%) than for avoided (selected <50%) directions (t(19) = 4.38, p < .001, Cohen’s d = .979; Figure 2d). Next, we investigated whether saccade costs predicted saccade selection on a trial-by-trial basis. To this end, we first determined the more affordable option for each trial using the established saccade cost map (Figure 1d). We predicted that participants would select the more affordable option. The more affordable option was chosen above chance level across participants (M = 56.64%, 95%-CI = [52.75%-60.52%], one-sample t-test against 50%: t(19) = 3.26, p = .004, Cohen’s d = .729; Figure 2e). This suggests that cost is rapidly weighed on a trial-by-trial basis during saccade selection, establishing that saccade costs predict saccade preferences.

2.3 Saccade costs predict saccade curvature and latency

If saccade cost is indeed weighed during saccade selection, this should be reflected in the oculomotor properties of the ensuing saccade. Saccade curvature reflects conflict between target and distractor saccade vectors: if a distractor is inhibited, and the target is activated, the saccade curves away from the distractor location [58–60]. Whenever saccade costs differ more between directions, there should therefore be more conflict between saccade vectors. If both directions were equally costly, there would be no need for conflict as cost minimization is impossible. We therefore hypothesized signs of increased oculomotor conflict to especially show in trials with relatively large differences in saccade costs. Furthermore, weighing costs (and reward) in decision-making is known to take time [61, 62]. More elaborate decisions should therefore not only show in more curvature, but also in longer saccade latencies.

To test this, we first split trials into saccades curving toward and away from the non-selected option. Saccades curved away from the non-selected option in the majority of trials, indicating oculomotor conflict (Figure 3a; M = 78.15%, 95%-CI = [74.734%-81.567%], t(19) = 15.741, p < .001, Cohen’s d = 3.519). We then examined how saccade curvature and latency predicted the difference in pupil size between the two possible saccade targets. Whenever the difference in pupil size between the two options was larger, saccades curved away more from the non-selected option (β = .004, SE = .001, t = 4.448, p < .001; Figure 3b), and their latencies slowed (β = .050, SE = .013, t = 4.323, p < .001; Figure 3c). Our results show that cost is actively weighed and leads to stronger conflict whenever cost differences are larger. This suggests that more elaborate decisions in saccade selection are predominantly made when warranted by sufficient differences in saccade costs between options.

Figure 3:

The above analyses show that saccade costs affect oculomotor conflict, but does increased conflict between saccade vectors also lead to selecting more affordable options? We expected that especially when oculomotor conflict was high, participants would choose the more affordable option. This means that saccade costs should be more predictive of saccade selection in away compared with toward curving trials. To test this idea, we repeated the trial-by-trial prediction of which option was selected as before (Figure 2e), but now separately for trials with toward and away curving saccades. Pupil size (i.e. saccade cost) predicted saccade selection when saccades curved away (M = 59.72%, 95%-CI = [55.208%-64.233%], t(19) = 4.116, p < .001, Cohen’s d = .920), but not toward (M = 48.42%, 95%-CI = [44.496%-52.338%], t(19) = .771, p = .450, Cohen’s d = .172) the non-selected option. These prediction accuracies differed between curve directions (t(19) = 4.795, p < .001, Cohen’s d = 1.072; Figure 3d). This shows that saccade costs were predominantly considered when saccades curved away. Together, these analyses suggest that the costs of potential saccade targets are especially weighed during saccade selection when warranted by large differences in saccade costs. In these cases, oculomotor conflict increases and saccade cost plays a bigger role in saccade selection.

2.4 Saccade costs predict saccade preferences in natural viewing

The previous results establish that saccade costs predict saccade preferences in highly controlled settings. However, a crucial question is whether saccade costs also predict saccade preferences in more complex and less controlled settings, in which physical saliency, the observer’s goals, and prior knowledge about the scene also affect saccade selection. To test this, we analyzed data from two existing datasets [63] wherein participants (total n = 41) searched for small targets (’Z’ or ‘H’) in natural scenes (Figure 4a; [64]). Again, we tested whether pupil size prior to saccades negatively linked with saccade preferences across directions. Because saccade costs and preferences across directions could differ for different situations, but should always be negatively linked, we established both cost and preferences independently in each dataset. Many factors influence pupil size in such a natural task, for which we controlled as much as possible by including variables known to covary with pupil size in a linear mixed-effects model (based on [63]; e.g. luminance, gaze position, saccade properties, saliency, fixation number; see Methods) to access the underlying saccade costs. As hypothesized, we observed a negative relationship between pupil size and saccade preferences in both experiments (Exp. 1: β = 1.784, SE = .324, t = 5.412, p < .001; saccade preferences in Figure 4b, link in Figure 4c; Exp. 2: β = .644, SE = .170, t = 3.780, p < .001; saccade preferences in Figure 4d, link in Figure 4e). This shows that even when participants made unconstrained eye movements in natural scenes, saccade cost remained linked to saccade preferences: affordable directions were preferred over costly directions.

Figure 4:

Do cognitive operations and eye movements consume from a similar pool of resources [49]? If so, increasing cognitive demand for non-oculomotor processes should result in decreasing available resources for the oculomotor system. In line with this idea, previous work indeed shows altered eye-movement behavior under effort as induced by dual tasks, for example by making less saccades under increased cognitive demand [65–67]. We therefore investigated whether less saccades were made as soon as participants performed an auditory digit-counting dual task in comparison to ignoring the auditory number stream (in Exp. 2; Figure 4a). Participants indeed reduced saccade frequency under the additional demand of the dual task (t(24) = 7.224, p < .001, Cohen’s d = 1.445; Figure 4h). This indicates that the auditory dual task and the oculomotor system, at least in part, consumed from a shared pool of cognitive resources.

From a costs-perspective, it should be efficient to not only adjust the number of saccades (non-specific), but also by cutting especially expensive directions the most (specific). Therefore, we expected participants to especially avoid costly saccades (as assessed in the single task) under higher cognitive demand (induced by the dual task). We calculated a saccade-adjustment map (Figure 4g) by subtracting the saccade preference map in the dual task (Figure 4f) from the single task map (Figure 4d). Participants seemingly cut vertical saccades in particular, and made more saccades to the top right direction. As hypothesized, pupil size negatively linked with the adjustment map (β = 9.333, SE = .966, t = 9.659, p < .001; Figure 4i; while controlling for the same possible covariates as before). This shows that costly saccades were cut disproportionally when more cognitive resources were consumed by the additional auditory dual task. This demonstrates that cognitive resources are flexibly (dis)allocated from and to the oculomotor system based on the current resource demands.

3 Discussion

We here investigated whether effort determines saccade preferences. We first measured pupil size prior to saccade execution across directions as a physiological marker of effort and thus saccade costs. Next, saccade preferences were assessed in the same participants and directions. We observed that affordable saccades were preferred over costly ones. When two possible saccade directions differed more in saccade cost, we found higher oculomotor conflict as indexed by stronger saccade trajectory deviations away from the non-selected option and increased onset latencies. In two additional experiments, we demonstrated the link between saccade costs and saccade preferences to be robust even when participants made unconstrained eye movements during natural viewing. Lastly, saccade directions were flexibly adjusted based on cost as cognitive demand increased. Together, this demonstrates that saccade costs fundamentally underlie saccade selection, even when physical salience, the goals of the observer, and selection history affect where the eyes are moved.

What contributes to saccade costs? We speculate that at least three processes contribute to the total cost of a saccade [30]: the complexity of oculomotor programming [22, 68, 69], shifting of presaccadic attention [70, 71], and predictive/spatial remapping [72–75]. The complexity of oculomotor programs is arguably shaped by its neural underpinnings as well as the number of muscles that are required to make certain saccades. For example, oblique saccades require more complex oculomotor programs than horizontal eye movements because more neuronal populations in superior colliculus (SC) and frontal eye fields (FEF) [76–79], and more muscles are necessary to plan and execute the saccade [76, 80, 81]. Besides the direction, other properties of the ensuing saccade such as its speed, distance, curvature and accuracy may contribute to the underlying saccade costs [22, 30, 55, 82, 83] but this remains to be investigated directly. Furthermore, presaccadic attention is shifted prior to each saccade to prepare the brain for the abrupt changes in retinal input resulting from saccades through spatial/predictive remapping [70–74, 84, 85]. This preparation for upcoming changes in retinal input consumes neurocognitive resources and therefore likely contributes to saccade costs. To better understand saccade selection more generally, future work should elucidate which processes contribute to saccade costs, and how costs shape different (aspects of) saccades.

The observed differences in saccade costs across directions could be linked to established anisotropies in perception [86–92], attention [93–97], and (early) visual cortex [98–102] [also see 103]. For example, downward saccades may be more costly than upward saccades due to the underrepresentation of the upper visual field in early visual areas [98–102] or due to stronger presaccadic benefits for upward compared with downward saccades [93, 94], but such suggestions remain to be tested directly. Future work should elucidate where saccade cost or the aforementioned anisotropies originate from and how they are related - something the present work cannot address.

How does the brain keep track of saccade costs, and which areas use it during saccade selection? Although our data do not allow direct inferences about the precise neural circuitry underlying the computations of oculomotor selection, oculomotor control is generally thought to be steered by a network encompassing the FEF [77, 104], the supplementary eye field [105–107], the anterior cingulate cortex [108–111], the SC [33, 112–117], and the cerebellum [58, 118–120]. These areas are not just associated with oculomotor control, but are all also thought to be crucial for decision-making processes [117, 121–131]. It is plausible that the weighing of saccade costs during saccade selection is performed by this oculomotor-decision making network, but other areas such as orbitofrontal cortex may also play a role [132, 133].

Throughout this paper, we have used cost in the limited context of saccades. However, every action, be it physical or cognitive, is associated with cost, and pupil size is likely a general marker of this [49]. Cost-based decision-making may therefore be a more general property of the brain [28, 44, 134]. For instance, we expect costs to similarly drive where to deploy covert attention. We here measured cost as the degree of LC- and effort-linked pupil dilation. Activity in LC with its widespread connections throughout the brain [31, 34–38] is considered to be crucial for the communication within and between neural populations and modulates global neural gain [40–43, 135]. Neural firing is costly [22, 136], and therefore LC activity and pupil size are (neuro)physiologically plausible markers of cost [33]. Tentative evidence even suggests that continued exertion of effort (accompanied by altered pupil dilation) is linked to the accumulation of glutamate in the lateral prefrontal cortex [137], which may be a metabolic marker of cost [also see 137, 138].

Besides the costs of increased neural activity under cognitive demand, effort should be considered costly for a second reason: Cognitive resources are limited. Therefore, any unnecessary resource expenditure reduces cognitive and behavioral flexibility [22, 28, 44]. As a result, the brain needs to distribute resources between cognitive operations and the oculomotor system. We found evidence for the idea that such tuning is adaptive, and flexibly adjusts based on the general level of cognitive demand and available resources: Increasing cognitive demand via an additional auditory dual task led to a lower saccade frequency, and especially costly saccades were cut. In other situations, more resources can be distributed to the oculomotor system, for example to discover new sources of reward [22, 139]. Adaptive tuning of resource allocation from, and to the oculomotor system parsimoniously explains a number of empirical observations. For example, higher cognitive demand is accompanied by smooth pursuits deviating more from to-be tracked targets [140], reduced (micro)saccade frequencies [Figure 4; 66, 67, 141, 142], and slower peak saccade velocities [143–145]. Together, we propose that cognitive resources are flexibly (dis)allocated to and from the oculomotor system based on the current demands to establish an optimal balance between performance and cost minimization whenever possible.

Saccade costs are likely relevant for decision-making beyond saccade selection itself. For instance, costs might affect how we shift attention (e.g. eye or head movements) and whether we choose to shift attention or use other cognitive operations such as working memory instead [30, 146–148]. Broadening the implications of our findings further, studying the computations underlying eye movements may allow to learn about decision-making more generally [22, 61, 124]. Where, how, and when to saccade must itself be considered the outcome of a decision [149]. As saccade selection is arguably the most frequent decision the brain is faced with [2, 3], the underlying decision process should therefore be highly optimized. The existence of such a well-tuned apparatus used for simple decisions may then be the basis for also more complex decisions. Understanding saccade selection may therefore have implications also for higher-level decision-making.

To conclude, we have demonstrated that saccade costs can be measured using pupil size and that these costs robustly predict saccade selection. We propose that saccade selection is driven by physical properties of the environment, the observer’s goals, selection history, and another fundamental factor: effort.

4 Methods

4.1 Saccade planning and saccade preference tasks

4.1.1 Participants

Twenty-two participants with normal or corrected-to-normal vision took part in the saccade planning and preference tasks across two sessions. One participant was excluded due to only finishing a single session, and another dataset was discarded due to not following task instructions (<50% included trials in the saccade planning task). Twenty participants were included in the analyses for the saccade planning, and saccade decision tasks (age: M = 24.00, range: [19–31], 12 women, 8 men). The current sample size was comparable with previous work investigating saccade costs [23, 30]. The total number of trials was substantially larger in the current dataset than in Koevoet et al. [30] (14,400 vs. 4,800), albeit from a slightly smaller number of participants (n = 20 vs. n = 24). Participants provided written informed consent before taking part, and were awarded monetary compensation or course credits. The experimental procedure was approved by Utrecht University’s ethical review board of the Faculty of Social Sciences (22-0635).

4.1.2 Apparatus and stimuli

Gaze position and pupil size were recorded at 1000 Hz with an Eyelink 1000 desktop mount (SR Research, Ontario, Canada) in a brightness- and sound-attenuated laboratory. A chin- and forehead-rest limited head movements. Stimuli were presented using PsychoPy [v.2022.2.5; 150] on an ASUS ROG PG278Q monitor (2560 x 1440, 100 Hz) positioned 67.5 cm away from eye position. The eye-tracker was calibrated (9 points) at the beginning of each session, during each break, and whenever necessary throughout the experiment (same procedure for both tasks, see below).

Potential saccade targets were eight equally spaced out red rings (1° diameter) positioned at an eccentricity of 10° visual angle. The central fixation stimulus was a red eight-legged asterisk of which each leg pointed towards one of the possible saccade targets (1°). These stimuli were presented on a blue circle (12° diameter; 11.64 cd/m²); the remaining part of the screen was black (0.73 cd/m²) to ensure equal brightness across all 36 possible target locations (Figure 1a, b).

To control for low-level visual effects on pupil size, the red color of all stimuli was made equiluminant to the blue background colour using a flicker fusion calibration [as in 30]. A blue background (HSV: 240.1.1) was presented continuously while a central red circle (5° diameter) continuously flickered at 25 Hz. Participants adjusted the luminance of the red color by moving the mouse across the horizontal plane of the screen until the flickering was the least noticeable, and then clicked the left mouse button to confirm. This procedure was performed thrice, and the average luminance of the red colour was used for the fixation and target stimuli throughout the task. Participants completed the flicker fusion calibration preceding each task.

4.1.3 Procedure

The experiment started with the saccade planning task [30], wherein participants planned saccades into 36 different cued directions. Each trial started when the central stimulus was fixated for 500 ms. After a fixation period (2000 ms) eight equally spaced potential saccade targets were presented (randomized which eight out of the 36 between trials). Afterwards, one of the eight legs of the asterisk became slightly thicker, cueing a saccade target (750-1250 ms, 100% valid). Participants planned and withheld an eye movement until cue offset, and then executed the saccade as fast as possible. Trials ended upon fixating the target stimulus (within 3°) for 500 ms. Whenever participants saccaded too early or to an incorrect location, red feedback text was presented (“too early”, “wrong location”). Each session consisted of 360 trials, preceded by ten practice trials. Participants could initiate a break whenever they wanted.

Participants subsequently completed a saccade preference task [adapted from 23]. Upon briefly fixating the central stimulus (10-500 ms), possible saccade targets were presented in two out of the 36 positions around the visual field. The only restriction was that the two targets should at least differ 20° in angle to ensure targets were sufficiently spaced out - and to limit saccade averaging [151]. Trials ended when one of the two saccade targets was fixated for 50 ms. Participants completed 360 trials per session.

4.1.4 Data processing and analyses

All data were analyzed using custom Python (v3.9.14) and R (v4.3.1) scripts. Analyses of pupillometric data followed recommendations by [33, 152]. Blinks were interpolated [152], data were downsampled to 100 Hz, and pupil data were subtractively baseline corrected with the mean of the first 250 ms after cue onset. Saccades were detected offline using an onset velocity threshold of 75°/s and an offset threshold of 1°/s. Trials with fast (<175 ms) or slow (>550 ms) onset latency [30], a very short (<10 ms) or long saccade (>110 ms) duration [153], an amplitude smaller than 5°, or saccades landing more than 2° from the target location, saccades toward the wrong target, and practice trials were discarded (13.67% in total). To map out saccade planning costs across directions, the average pupil size was calculated 150 ms before until 170 ms after cue offset (Figure 1f) - before any saccade onsets to prevent pupil foreshortening errors [154]. We analyzed saccade costs by incorporating continuous predictors for oblique (cardinal vs. oblique; 0-4), vertical (up vs. down; in y coordinates) and horizontal (left vs. right; in x coordinates) direction biases in a linear mixed-effects model (Figure 1e, f). We also incorporated properties of the ensuing saccade to control for their possible associations with pupil size. The final model was determined using AIC-based backward model selection (Wilkinson notation: pupil size ∼ oblique*saccade duration + vertical + horizontal + amplitude + landing error + peak velocity + (1 + oblique + vertical|participant)). For all mixed-effects models, we included as many by-participant random slopes as possible for our main effects of interest while ensuring model convergence [63, 155]. To obtain an average saccade costs map, pupil sizes were z-transformed per participant within sessions, and then averaged across participants for each direction (Figure 1d).

For the saccade preference task, saccades were detected as above. Which saccade target was selected per trial was determined using the last 50 ms of gaze data of each trial - the option closest to the gaze position was treated as the selected target. Trials were discarded if the difference in distance between the two saccade options in gaze position was less than 1.5° (3.47%). A logistic mixed-effects model was fit to investigate anisotropies across directions (Wilkinson notation: saccade preference ∼ oblique + vertical + horizontal + (1 + oblique + vertical + horizontal|participant)). Saccade preference for each direction was calculated per participant by summing how often a direction was chosen and then dividing by the number of times that direction was offered. As for the saccade cost map, the average saccade preference map was obtained by averaging across participants (Figure 2b). Preferred (>50%) and avoided (<50% chosen) directions were grouped using the average preference map (Figure 2d). To investigate if cost predicted saccade selection on a trial-by-trial basis, we compared the saccade costs of the two potential options. We predicted that the option with the smaller pupil size from the average cost map (obtained from the saccade planning task) would be chosen. This procedure was performed for each participant, and subsequently tested against chance performance (50%) with a one-sample t-test.

Saccade curvature was computed using the peak deviation from a straight line between gaze position at saccade onset until saccade offset [Figure 3a; 58]. Trials were excluded from these analyses if: a) saccade latencies were shorter than 175 ms, b) saccade amplitudes were smaller than 5°, c) saccade durations were shorter than 10 ms or longer than 110 ms and d) if the angle between targets exceeded 150° [30, 151, 153]. To analyze the relationship between saccade costs and saccade properties, we first computed the absolute saccade cost difference for each trial (as indexed from the average saccade cost map). A linear mixed-effects model was conducted to test whether saccade curvature and latency linked to saccade costs (Wilkinson notation: cost difference ∼ peak deviation + latency + (1 + peak deviation|participant)). We then split the data based on whether saccades curved toward or away from the non-selected option. The same trial-by-trial analysis as described above was used to investigate if cost predicted saccade selection in toward and away trials separately.

4.2 Search in natural scenes

4.2.1 Procedure

We analyzed existing data of two experiments to investigate if effort drives saccade selection in a more natural task [for an exhaustive explanation of the procedure see the original paper 63]. Briefly, in Experiments 1 and 2, sixteen and twenty-five participants respectively searched for small letters (’Z’ or ‘H’) in natural scenes (from [64]). As in the saccade planning and preference tasks, gaze position and pupil size were recorded with an Eyelink 1000 (SR Research, Ontario, Canada) at 1000 Hz. Stimuli were presented (1280 x 1024) using OpenSesame [156] with the PsychoPy backend [150].

In Experiment 2, auditory digits (0-9) were presented with an inter-digit interval of 1500 ms during search - note that Experiment 1 did not feature the auditory dual task. Crucially, participants either performed a dual task wherein the count of a specific digit was monitored throughout search, or a single task where the number stream was ignored. The single and dual conditions were blocked, and the sequence of these blocks was random across participants.

4.2.2 Data processing and analyses

Pupil size was averaged per fixation and subsequently z-transformed per participant [63]. Fixations with pupil sizes deviating more than 3SD from the mean (within a participant) and fixations positioned outside of the monitor were excluded to mitigate possible confounds (Exp. 1: 4.32%, Exp. 2: 9.75% discarded). 57,127 and 214,449 fixations were analyzed from Experiments 1 and 2, respectively. Fixations were classified into 36 bins based on their direction (bins consistent with the saccade planning and preference tasks).

To investigate if saccade costs predicted saccade preferences when searching in natural scenes, we analyzed all fixations from Experiment 1, and fixations from the single condition in Experiment 2. Next, we computed the average saccade preference map separately for each experiment by calculating the percentage of saccades in any of the 36 directions. Linear mixed-effects models were used to investigate whether this preference map predicted pupil size on a fixation-by-fixation basis in both experiments. We controlled for as many possible factors that are known to covary with pupil size in our model to control for them as much as possible to attempt to access the underlying saccade cost signal [63]; Wilkinson notation: pupil size ∼ saccade preferences + luminance + saliency + fixation number + trial number + x gaze coordinate + y gaze coordinate + saccade duration + fixation duration + saccade amplitude + (1 + saccade preferences|participant).

To investigate if costly saccades were avoided in particular when the overall level of demand increased via the dual task, we analyzed data from Experiment 2, The percentages of saccades made into each direction for the single and dual conditions were calculated. We subtracted these averaged preference maps to obtain an adjustment map: this revealed how participants altered their saccade preferences under additional demand (Figure 4e). We predicted pupil size using the average adjustment map for each direction while again controlling for many possible confounding factors in the single condition using a linear mixed-effects model on a fixation-by-fixation basis (Wilkinson notation: pupil size ∼ saccade adjustment + luminance + saliency + fixation number + trial number + x gaze coordinate + y gaze coordinate + saccade duration + fixation duration + saccade amplitude + (1 + saccade adjustment|participant)).

Acknowledgements

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement n° 863732).

Declaration of interest

The authors declare no conflicting interests.

Data availability and code availability

Data and analyses scripts to reproduce the results are available via the Open Science Framework: https://osf.io/n3ktm/?view_only=6037c94b990e4bde8512918aca043e0a.

Supporting Information

Supporting Figure 1:

Supporting Table 1:

Supporting Table 2:

Supporting Table 3:

Supporting Table 4: