Effort Drives Saccade Selection

Reviewer #1 (Public Review):

Vision is a highly active process. Humans move their eyes 3-4 times per second to sample information with high visual acuity from our environment, and where eye movements are directed is critical to our understanding of active vision. Here, the authors propose that the cost of making a saccade contributes critically to saccade selection (i.e., whether and where to move the eyes). The authors build on their own recent work that the effort (as measured by pupil size) that comes with planning and generating an eye movement varies with saccade direction. To do this, the authors first measured pupil size for different saccade directions for each participant. They then correlated the variations in pupil size obtained in the mapping task with the saccade decision in a free-choice task. The authors observed a striking correlation: pupil size in the mapping task predicted the decision of where to move the eyes in the free choice task. In this study, the authors provide a number of additional insightful analyses (e.g., based on saccade curvature, and saccade latency) and experiments that further support their claim that the decision to move the eyes is influenced by the effort to move the eyes in a particular direction. One experiment showed that the same influence of assumed saccade costs on saccade selection is observed during visual search in natural scenes. Moreover, increasing the cognitive load by adding an auditory counting task reduced the number of saccades, and in particular reduced the costly saccades. In sum, these experiments form a nice package that convincingly establishes the association between pupil size and saccade selection.

In my opinion, the causal structure underlying the observed results is not so clear. While the relationship between pupil size and saccade selection is compelling, it is not clear that saccade-related effort (i.e., the cost of a saccade) really drives saccade selection. Given the correlational nature of this relationship, there are other alternatives that could explain the finding. For example, saccade latency and the variance in landing positions also vary across saccade directions. This can be interpreted for instance that there are variations in oculomotor noise across saccade directions, and maybe the oculomotor system seeks to minimize that noise in a free-choice task. In fact, given such a correlational result, many other alternative mechanisms are possible. While I think the authors' approach of systematically exploring what we can learn about saccade selection using pupil size is interesting, it would be important to know what exactly pupil size can add that was not previously known by simply analyzing saccade latency. For example, saccade latency anisotropies across saccade directions are well known, and the authors also show here that saccade costs are related to saccade latency. An important question would be to compare how pupil size and saccade latency uniquely contribute to saccade selection. That is, the authors could apply the exact same logic to their analysis by first determining how saccade latencies (or variations in saccade landing positions; see Greenwood et al., 2017 PNAS) vary across saccade directions and how this saccade latency map explains saccade selection in subsequent tasks. Is it more advantageous to use one or the other saccade metric, and how well does a saccade latency map correlate with a pupil size map?

In addition to eye-movement-related anisotropies across the visual field, there are of course many studies reporting visual field anisotropies (see Himmelberg, Winawer & Carrasco, 2023, Trends in Neuroscience for a review). It would be interesting to understand how the authors think about visual field anisotropies in the context of their own study. Do they think that their results are (in)dependent on such visual field variations (see Greenwood et al., 2017, PNAS; Ohl, Kroell, & Rolfs, 2024, JEP:Gen for a similar discussion)?

Finally, the authors conclude that their results "suggests that the eye-movement system and other cognitive operations consume similar resources that are flexibly allocated among each other as cognitive demand changes. The authors should speculate what these similar resources could mean? What are the specific operations of the auditory task that overlap in terms of resources with the eye movement system?

Reviewer #2 (Public Review):

The authors attempt to establish presaccadic pupil size as an index of 'saccade effort' and propose this index as one new predictor of saccade target selection. They only partially achieved their aim: When choosing between two saccade directions, the less costly direction, according to preceding pupil size, is preferred. However, the claim that with increased cognitive demand participants would especially cut costly directions is not supported by the data. I would have expected to see a negative correlation between saccade effort and saccade direction 'change' under increased load. Yet participants mostly cut upwards saccades, but not other directions that, according to pupil size, are equally or even more costly (e.g. oblique saccades).

Strengths:

The paper is well-written, easy to understand, and nicely illustrated.

The sample size seems appropriate, and the data were collected and analyzed using solid and validated methodology.

Overall, I find the topic of investigating factors that drive saccade choices highly interesting and relevant.

Weaknesses:

The authors obtain pupil size and saccade preference measures in two separate tasks. Relating these two measures is problematic because the computations that underly saccade preparation differ. In Experiment 1, the saccade is cued centrally, and has to be delayed until a "go-signal" is presented; In Experiment 2, an immediate saccade is executed to an exogenously cued peripheral target. The 'costs' in Experiment 1 (computing the saccade target location from a central cue; withholding the saccade) do not relate to Experiment 2. It is unfortunate, that measuring presaccadic pupil size directly in the comparatively more 'natural' Experiment 2 (where saccades did not have to be artificially withheld) does not seem to be possible. This questions the practical application of pupil size as an index of saccade effort

The authors claim that the observed direction-specific 'saccade costs' obtained in Experiment 1 "were not mediated by differences in saccade properties, such as duration, amplitude, peak velocity, and landing precision (Figure 1e,f)". Saccade latency, however, was not taken into account here but is discussed for Experiment 2.

The apparent similarity of saccade latencies and pupil size, however, is striking. Previous work shows shorter latencies for cardinal than oblique saccades, and shorter latencies for horizontal and upward saccades than downward saccades - directly reflecting the pupil sizes obtained in Experiment 1 as well as in the authors' previous study (Koevoet et al., 2023, PsychScience).

-

The authors state that "from a costs-perspective, it should be efficient to not only adjust the number of saccades (non-specific), but also by cutting especially expensive directions the most (specific)". However, saccade targets should be selected based on the maximum expected information gain. If cognitive load increases (due to an additional task) an effective strategy seems to be to perform less - but still meaningful - saccades. How would it help natural orienting to selectively cut saccades in certain (effortful) directions? Choosing saccade targets based on comfort, over information gain, would result in overall more saccades to be made - which is non-optimal, also from a cost perspective.

Overall, I am not sure what practical relevance the relation between pupil size (measured in a separate experiment) and saccade decisions has for eye movement research/vision science. Pupil size does not seem to be a straightforward measure of saccade effort. Saccade latency, instead, can be easily extracted in any eye movement experiment (no need to conduct a separate, delayed saccade task to measure pupil dilation), and seems to be an equally good index.

Reviewer #3 (Public Review):

This manuscript extends previous research by this group by relating variation in pupil size to the endpoints of saccades produced by human participants under various conditions including trial-based choices between pairs of spots and search for small items in natural scenes. Based on the premise that pupil size is a reliable proxy of "effort", the authors conclude that less costly saccade targets are preferred. Finding that this preference was influenced by the performance of a non-visual, attention-demanding task, the authors conclude that a common source of effort animates gaze behavior and other cognitive tasks.

Strengths:

Strengths of the manuscript include the novelty of the approach, the clarity of the findings, and the community interest in the problem.

Weaknesses:

Enthusiasm for this manuscript is reduced by the following weaknesses:

(1) A relationship between pupil size and saccade production seems clear based on the authors' previous and current work. What is at issue is the interpretation. The authors test one, preferred hypothesis, and the narrative of the manuscript treats the hypothesis that pupil size is a proxy of effort as beyond dispute or question. The stated elements of their argument seem to go like this:
PROPOSITION 1: Pupil size varies systematically across task conditions, being larger when tasks are more demanding.
PROPOSITION 2: Pupil size is related to the locus coeruleus.
PROPOSITION 3: The locus coeruleus NE system modulates neural activity and interactions.
CONCLUSION: Therefore, pupil size indexes the resource demand or "effort" associated with task conditions.
How the conclusion follows from the propositions is not self-evident. Proposition 3, in particular, fails to establish the link that is supposed to lead to the conclusion.

(2) The authors test one, preferred hypothesis and do not consider plausible alternatives. Is "cost" the only conceivable hypothesis? The hypothesis is framed in very narrow terms. For example, the cholinergic and dopamine systems that have been featured in other researchers' consideration of pupil size modulation are missing here. Thus, because the authors do not rule out plausible alternative hypotheses, the logical structure of this manuscript can be criticized as committing the fallacy of affirming the consequent.

(3) The authors cite particular publications in support of the claim that saccade selection is influenced by an assessment of effort. Given the extensive work by others on this general topic, the skeptic could regard the theoretical perspective of this manuscript as too impoverished. Their work may be enhanced by consideration of other work on this general topic, e.g, (i) Shenhav A, Botvinick MM, Cohen JD. (2013) The expected value of control: an integrative theory of anterior cingulate cortex function. Neuron. 2013 Jul 24;79(2):217-40. (ii) Müller T, Husain M, Apps MAJ. (2022) Preferences for seeking effort or reward information bias the willingness to work. Sci Rep. 2022 Nov 14;12(1):19486. (iii) Bustamante LA, Oshinowo T, Lee JR, Tong E, Burton AR, Shenhav A, Cohen JD, Daw ND. (2023) Effort Foraging Task reveals a positive correlation between individual differences in the cost of cognitive and physical effort in humans. Proc Natl Acad Sci U S A. 2023 Dec 12;120(50):e2221510120.

(4) What is the source of cost in saccade production? What is the currency of that cost? The authors state (page 13), "... oblique saccades require more complex oculomotor programs than horizontal eye movements because more neuronal populations in the superior colliculus (SC) and frontal eye fields (FEF) [76-79], and more muscles are necessary to plan and execute the saccade [76, 80, 81]." This statement raises questions and concerns. First, the basis of the claim that more neurons in FEF and SC are needed for oblique versus cardinal saccades is not established in any of the publications cited. Second, the authors may be referring to the fact that oblique saccades require coordination between pontine and midbrain circuits. This must be clarified. Second, the cost is unlikely to originate in extraocular muscle fatigue because the muscle fibers are so different from skeletal muscles, being fundamentally less fatigable. Third, if net muscle contraction is the cost, then why are upward saccades, which require the eyelid, not more expensive than downward? Thus, just how some saccades are more effortful than others is not clear.

(5) The authors do not consider observations about variation in pupil size that seem to be incompatible with the preferred hypothesis. For example, at least two studies have described systematically larger pupil dilation associated with faster relative to accurate performance in manual and saccade tasks (e.g., Naber M, Murphy P. Pupillometric investigation into the speed-accuracy trade-off in a visuo-motor aiming task. Psychophysiology. 2020 Mar;57(3):e13499; Reppert TR, Heitz RP, Schall JD. Neural mechanisms for executive control of speed-accuracy trade-off. Cell Rep. 2023 Nov 28;42(11):113422). Is the fast relative to the accurate option necessarily more costly?

(6) The authors draw conclusions based on trends across participants, but they should be more transparent about variation that contradicts these trends. In Figures 3 and 4 we see many participants producing behavior unlike most others. Who are they? Why do they look so different? Is it just noise, or do different participants adopt different policies?

Author Response:

We appreciate the thorough comments from the reviewers. Before revising the manuscript, we would like to briefly reply to the main concerns raised:

• Is pupil size a reliable proxy of effort? A vast amount of work demonstrates that pupil size sensitively scales with fluctuations in effort: for instance, the pupil dilates when increasing load in working memory, or multiple object tracking tasks, and such pupillary effects robustly explain individual differences in cognitive ability and fluctuations in performance across trials.1–4 This extends to the planning of movements as pupil dilations are observed prior to the execution of (eye) movements.5 As reviewed previously6–12 (based on vast literature each), any increase in effort is associated with an increase in pupil size. Inadvertently, we phrased as if the link between effort and pupil size was established via shared neural correlates. However, this is not the case as the link between effort and pupil size had been established well before the underlying neural circuitry of this relationship was investigated in detail. During the revision, we plan to rewrite this section to clarify that pupil size indexes effort and to provide a clear distinction between this link and putative neural underpinnings of such effort-linked modulations.

• Is saccade latency an alternative explanation for the link between effort and saccade selection? Longer saccade latencies may imply more complex oculomotor programming (e.g. saccades with larger amplitudes require longer latencies for non-microsaccades13, and latencies increase when distractors are presented14), and latencies are indeed known to differ across directions15,16. As suggested, it is possible that saccade latencies may also predict saccade preferences. However, even if this is the case, this would not constitute an alternative explanation. As saccade latency may index oculomotor programming complexity, it can potentially be considered an alternative outcome measure of effort, albeit restricted to the context of saccades. Therefore, if saccade latencies predict saccade preferences, this would not affect our conclusion, rather it would constitute as converging evidence that supports the conclusion that effort drives saccade selection.

A related question is why one would use pupil size as a measure of effort, given the methodological care that pupillometry requires. There are a number of points that make pupil size sensible and promising in comparison with saccade latencies. In contrast to saccade latencies, pupil size allows to capture the effort of different effector systems (e.g. head or hand movements), and potentially even the effort associated with covert shifts of attention. Moreover, pupil size is a temporally rich and continuous measure that allows to isolate processes unfolding prior to (eye) movement onset (e.g. oculomotor programming). Together, this makes pupil size a powerful tool to study the costs of visual selection more broadly. In the revision, we will add analyses incorporating latencies and other other saccade metrics. We will also discuss the differences between pupil size and saccade latencies in capturing saccade costs and effort.

• Are the current results causal or correlational? Most of the currently reported results are indeed correlational in nature. In our first tasks, we correlated pupil size during saccade planning to saccade preferences in a subsequent task. Although the link between across tasks was correlational, the observed relationship clearly followed our previously specified hypothesis.17 Moreover, experiments 1 and 2 of the visual search data replicated and extended this relationship. We also directly manipulated cognitive demand in the second visual search experiment. In line with the hypothesis that effort affects saccade selection, participants executed less saccades overall when performing a (primary) auditory dual task, and even cut the costly saccades most. Whilst mostly correlational, we do not know of a more fitting and parsimonious explanation for our findings than effort predicting saccade selection. We will address causality in the discussion for transparency and point more clearly to the second visual search experiment for causal evidence.

References

(1) Alnæs, D. et al. Pupil size signals mental effort deployed during multiple object tracking and predicts brain activity in the dorsal attention network and the locus coeruleus. J. Vis. 14, 1 (2014).

(2) Koevoet, D., Strauch, C., Van der Stigchel, S., Mathôt, S. & Naber, M. Revealing visual working memory operations with pupillometry: Encoding, maintenance, and prioritization. WIREs Cogn. Sci. e1668 (2023) doi:10.1002/wcs.1668.

(3) Robison, M. K. & Unsworth, N. Pupillometry tracks fluctuations in working memory performance. Atten. Percept. Psychophys. 81, 407–419 (2019).

(4) Unsworth, N. & Miller, A. L. Individual Differences in the Intensity and Consistency of Attention. Curr. Dir. Psychol. Sci. 30, 391–400 (2021).

(5) Richer, F. & Beatty, J. Pupillary Dilations in Movement Preparation and Execution. Psychophysiology 22, 204–207 (1985).

(6) Bumke, O. Die Pupillenstörungen Bei Geistes-Und Nervenkrankheiten. (Fischer, 1911).

(7) Kahneman, D. Attention and Effort. (Prentice-Hall, 1973).

(8) van der Wel, P. & van Steenbergen, H. Pupil dilation as an index of effort in cognitive control tasks: A review. Psychon. Bull. Rev. 25, 2005–2015 (2018).

(9) Loewenfeld, I. E. Mechanisms of reflex dilatation of the pupil. Doc. Ophthalmol. 12, 185–448 (1958).

(10) Mathôt, S. Pupillometry: Psychology, Physiology, and Function. J. Cogn. 1, 16 (2018).

(11) Sirois, S. & Brisson, J. Pupillometry. WIREs Cogn. Sci. 5, 679–692 (2014).

(12) Strauch, C., Wang, C.-A., Einhäuser, W., Van der Stigchel, S. & Naber, M. Pupillometry as an integrated readout of distinct attentional networks. Trends Neurosci. 45, 635–647 (2022).

(13) Kalesnykas, R. P. & Hallett, P. E. Retinal eccentricity and the latency of eye saccades. Vision Res. 34, 517–531 (1994).

(14) Walker, R., Deubel, H., Schneider, W. X. & Findlay, J. M. Effect of Remote Distractors on Saccade Programming: Evidence for an Extended Fixation Zone. J. Neurophysiol. 78, 1108–1119 (1997).

(15) Hanning, N. M., Himmelberg, M. M. & Carrasco, M. Presaccadic attention enhances contrast sensitivity, but not at the upper vertical meridian. iScience 25, 103851 (2022).

(16) Hanning, N. M., Himmelberg, M. M. & Carrasco, M. Presaccadic Attention Depends on Eye Movement Direction and Is Related to V1 Cortical Magnification. J. Neurosci. 4

4, (2024).

(17) Koevoet, D., Strauch, C., Naber, M. & Van der Stigchel, S. The Costs of Paying Overt and Covert Attention Assessed With Pupillometry. Psychol. Sci. 34, 887–898 (2023).