Eye movements are involved in almost every daily human activity, from searching for your apartment key, identifying a friend, reading, and preparing a sandwich. People make about three ballistic eye movements (saccades) per second orienting the central part of the vision (the fovea) to regions of interest in the world and acquiring high-acuity visual information (Bahill et al., 1975). A foveated visual system allocates more retinal cells (Curcio et al., 1987), thalamic, cortical (Azzopardi and Cowey, 1993), and superior colliculus (Chen et al., 2019) neurons to the central part of the visual field and allows for computational and metabolic savings. But it requires that eye movements are programmed intelligently to overcome the deficits of peripheral processing. People execute eye movements effortlessly, rapidly, and automatically giving the impression that these might seem fairly random. However, there are sophisticated computations in the brain that control eye movements involved in fine spatial judgments (Intoy and Rucci, 2020; Rucci et al., 2007), search (Araujo et al., 2001; Eckstein et al., 2015; Hoppe and Rothkopf, 2019; Najemnik and Geisler, 2005), object identification (Renninger et al., 2007), face recognition (Or et al., 2015; Peterson and Eckstein, 2012), reading (Legge et al., 1997; Legge et al., 2002), and motor actions (Ballard et al., 1995; Hayhoe and Ballard, 2005).

Following the gaze of others (gaze-following) with eye movements is critical to infer others’ intentions, current interests, and future actions (Capozzi and Ristic, 2018; Dalmaso et al., 2020; Emery, 2000; Kleinke, 1986; McKay et al., 2021). Gaze-following behavior can be observed as early as 8–10 months in infants and is widely found in nonhumans such as macaques, dogs, and goats (Brooks and Meltzoff, 2005; Kaminski et al., 2005; Senju and Csibra, 2008; Shepherd, 2010; Wallis et al., 2015). The ability to perceive others’ gaze direction accurately and plan eye movements is essential for infants to engage in social interactions to learn objects and languages (Carpenter et al., 1998; Morales et al., 1998; Morales et al., 2000; Woodward, 2003). People are extremely sensitive to others’ direction of gaze (Kleinke, 1986; Langton and Bruce, 1999). When people observe someone’s gaze, they orient covert attention and eye movements toward the gazed location, which improves the detection of a target appearing in the gaze direction (Driver et al., 1999; Egeth and Yantis, 1997; Friesen et al., 2004; Han et al., 2021b; Jonides, 1981; Kingstone et al., 2003; Mulckhuyse and Theeuwes, 2010). Impairments in gaze cueing have also been proposed as an important correlate of autism spectrum disorder (Baron-Cohen, 2001; Leekam et al., 1998; Nation and Penny, 2008) and important in child development (Brooks and Meltzoff, 2005).

The majority of studies investigating gaze-following (but see Gregory, 2022; Han et al., 2021b; Lachat et al., 2012; Macdonald and Tatler, 2013; Sun et al., 2017; Wang et al., 2014) use static images of the eyes or the face in isolation, which are far from the more ecological real-world behaviors of individuals moving their heads and eyes when orienting attention. That gaze cueing triggers eye movements is well known, but the dynamics of eye movements when observing gaze behaviors with naturalistic dynamic stimuli are not known. Studies have investigated how the brain integrates temporal information to program saccades and how it integrates foveal and peripheral information (Stewart et al., 2020; Wolf et al., 2022; Wolf and Schütz, 2015) but have relied on artificial or simplified stimuli.

Little is known about what features across the visual field influence eye movements during gaze-following, their temporal dynamics, and their functionality. How does the brain rely on the features of the gazer’s head and peripheral visual information about likely gaze goals to program eye movements? Do observers wait for the gazer’s head movement to end before initiating the first gaze-following saccades? Do visual properties of the gazer’s head influence the programming of eye movements? Answering these questions has been difficult because they require a well-controlled real-world data set, moment-to-moment characterization of the gazer’s features, and experimental manipulations that alter peripheral information while maintaining the gazer’s information unaltered.

Here, we created a collection of in-house videos of dynamic gaze behaviors in real-world settings by instructing actors to direct their gaze to specific people on the filming set (Figure 1a). We then asked experiment participants that watched the videos to follow the gaze shifts in the videos and decide whether a specific target person was present or absent (Figure 1a). The viewing angles subtended by the people in the videos corresponded to distances for which the eyes provided little information. Thus, our work focuses on the gazer’s head direction.

Figure 1 with 1 supplement see all

Download asset Open asset

Our first goal is to assess the impact of peripheral gaze-goal information on the saccade error and timing. Second, we aimed to elucidate how the brain temporally processes visual information to influence saccade programming during gaze-following. To extract features of the videos that we hypothesized would influence saccade planning we used a state-of-the-art artificial intelligence (AI) model (Chong et al., 2020) to make moment-to-moment estimates of the gazer’s head direction in the videos. We assessed how observers’ saccade direction, timing, and errors related to the extracted features to gain insight into the brain computations during saccade planning.

We investigated eye movement control while following the gaze of others. Although human eye movements are fast and might seem idiosyncratic, our findings show that the human brain uses moment-to-moment information about the gazer’s head dynamics and peripheral information about likely gaze goals to rationally plan the timing and endpoint of saccadic eye movements.

First, we found that the oculomotor system integrates information about the foveally presented gazer’s head and peripheral information about potential gaze goals. When a person was present at the gaze goal, observers executed faster and more accurate saccades. One might have expected the presence of gaze goals to reduce the saccade endpoint Euclidean error but to a lesser degree reduce saccade angular error. Arguably, if observers waited for the gazer’s head to stop, they could rely on the head direction to accurately plan their gaze-following saccade direction without needing to rely on a peripheral gaze goal. Our results showed otherwise. Deleting the gaze goal also increased the saccade angular error. This suggests that the eye movement direction also heavily relies on the peripheral processing of likely gaze goals. This might seem counterintuitive but if one considers that the saccades are programmed before the gazer’s head comes to a stop and points to the gaze goal, then the gazer’s head direction might not provide sufficient information for observers to program saccades to the correct direction.

The evidence for the integration of foveal and peripheral information is consistent with a series of studies showing observers’ ability to simultaneously process foveal and peripheral information for simpler dual tasks with simple stimuli (Ludwig et al., 2014; Stewart et al., 2020) and their joint influence on fixation duration during scene viewing (Laubrock et al., 2013) and subsequent eye movements (Wolf et al., 2022).

Importantly, we found the first saccades are anticipatorily initiated before the gazer’s head movement comes to a stop. And they contain information about the direction of the gaze goal that is more accurate than the direction information provided by the gazer’s head at the time of saccade initiation. This suggests that the brain is using peripheral information to make an active prediction about likely gaze goals. We found that on average the first saccades were anticipating the head direction by 340–370ms and 160–220ms for target present and absent conditions. Furthermore, previous studies have shown that a saccade is typically based on visual information presented ~100ms before saccade execution (Becker and Jürgens, 1979; Caspi et al., 2004; Hooge and Erkelens, 1999; Ludwig et al., 2005). Thus, the first saccade might only have access to the gazer’s head direction up to ~100ms before saccade execution, which means the observes’ first saccades might be anticipating the gazer’s head direction by 440–470ms and 260–320ms. Similarly, the gazer’s head direction at the time of saccade programming (rather than execution) was, on average, pointing 44.1 degrees (for target/distractor present trials) and 39.7 degrees (for target/distractor absent trials) away from the gaze goal.

The evidence for an inferential process that influences saccade programming when a person is present as the gaze goal in the scene might be expected. But, the inferential process still prevailed when a target/distractor was absent. It is likely that even when no person is present at the gaze goal location, the brain uses information about the scene including the ground, the objects, and the sky to make estimates of likely gaze goals. Prior knowledge about the maximum angular rotation of the gazer’s head also constrains the possible gazer goals. This information is used by observers to program anticipatory inferential eye movements even in the absence of an unambiguous visible gazer goal (e.g., a person) in the observer’s visual periphery.

Second, we found that early first saccades are executed when the gazer’s head velocity diminishes to values comparable to the velocity that is typical during the 200ms time interval before the head stops. This is consistent with the idea that observers use the gazer’s head velocity to dynamically make inferences about the likelihood that the head will stop and then execute a saccade towards the likely gaze goals. However, our data also suggest that other cues are used to infer that the gazer’s head will stop. For example, for some trials with longer first saccade latencies (no reverse saccade trials), the head velocity before the observers’ saccade execution is almost double the typical head velocities during the 200ms time interval before a head stops (Figure 4e). Thus, the observers must rely on other cues. In these long latency trials (Figure 4e) there is a reduction of the head velocity in the 200ms before the saccade execution suggesting that observers use the head’s deceleration to infer that the gazer’s head will come to a stop and then execute the first gaze-following saccade. It is also likely that for trials with a gaze goal, observers use an estimated error between the implied gaze direction and the gaze goal to plan saccades. Small estimated angular errors might be used to trigger saccades. Thus, we suggest that the observers’ oculomotor system might use multiple cues (head velocity, head deceleration, estimated gaze errors, etc) to infer that the gazer’s head might stop and trigger gaze-following saccades.

Third, surprisingly, we found that observers often executed reverse saccades in a significant proportion of trials (>20%). These reverse saccades are directed to re-fixate the gazer’s head and can be referred to as return saccades. The reverse saccades were not an artifact of our instruction to the observers to follow the gaze of the person in the video. A follow-up experiment where observers were instructed to decide whether a target person was present with no instruction to participants about their eye movements also resulted in a comparable proportion of reverse saccades. Why might observers make such saccades? Our analysis showed that these reverse saccades do not appear randomly across trials. Reverse saccades occur on trials in which the gazer’s head velocity is slow but starts accelerating about 200ms before the first saccade is executed and observers infer that the gazer’s head will not come to a halt. Why don’t observers simply cancel the forward saccade? Studies have shown that there is a 50–100ms delay between the programming of a saccade and its execution (Becker and Jürgens, 1979; Caspi et al., 2004; Ludwig et al., 2005). The gazer’s head acceleration occurring immediately before the execution of the forward saccade is not used to cancel the impending planned eye movement.

Our findings with the experiment that freezes the gazer after the first forward saccade suggest that the reverse saccade is programmed before the execution of the first forward saccade. This concurrent programming of saccades has been documented for simplified lab experiments (Becker and Jürgens, 1979; Caspi et al., 2004) but not in the context of real-world stimuli and tasks. One alternative explanation we did not explore is that reverse saccades are simply triggered after the first forward saccades that do not land on the target/distractor. In this perspective, a forward saccade is executed and when foveal processing determines that the saccade endpoint was far from a likely gaze goal then a reverse saccade is programmed and executed (regardless of the velocity of the gazer’s head). Data analysis does not favor this interpretation. In a small percentage of trials (15% of reverse saccade trials) first saccades landed within 0.5° visual angle of the target but these were still followed by reverse saccades. This observation suggests that foveating close to the gaze goal was not sufficient to interrupt a reverse saccade programmed before the execution of the first forward saccade.

Our results also show that the reverse saccades had functional importance. Forward saccades, after re-fixating the gazer with a reverse saccade, were more accurate at landing close to the gaze goal. The benefit of re-fixating the gazer was more reliable when there was a person present at the gaze goal. When a gaze goal person was absent we found a less reliable re-fixation benefit (not statistically significant in experiment 1 and marginally significant in experiment 2) suggesting that not having a peripheral likely gaze goal can be a bottleneck to the accuracy of saccade endpoints.

The existence of reverse saccades to re-fixate the gazer’s head might seem puzzling. Why does the oculomotor programming system not wait longer until the gazer’s head comes to a full stop, then executes the gaze-following saccade and avoids programming reverse saccades altogether? Executing anticipatory eye movements that predict future grasping actions (Mennie et al., 2007; Pelz and Canosa, 2001), the location or motion of a stimulus (Fooken and Spering, 2020; Kowler, 1989; Kowler, 2011; Kowler et al., 2019) is common for the oculomotor system. Thus, while following the gaze of others the observers’ oculomotor system plans anticipatory saccades that predict the gaze goal before the completion of the gazer’s head movements. Occasionally, these predictive saccades are premature and the brain rapidly programs a reverse saccade to re-fixate the gazer and collect further information about the potential gazer goal.

Are the reverse saccades unique to gaze following? No, humans make reverse saccades in other visual tasks that require maintaining information in working memory, such as copying a color block pattern across two locations (Hayhoe and Ballard, 2005; Hayhoe, 2017; Meghanathan et al., 2019). Most notably during reading humans make frequent reverse saccades (“called regressive”). Although one might draw a parallel between reading and gaze-following, our findings highlight important distinctions. Regressive saccades during reading are related to inaccurate eye movements that missed critical words or fixations that are too short to deeply process a word’s meaning (Inhoff et al., 2019; Rayner, 1998). The reverse saccades while following dynamic gaze are related to moment-to-moment changes in the visual information in the world (i.e. the gazer’s head velocity) and the oculomotor systems’ rapid response to optimize gaze-following.

A remaining question is whether the gaze-following behaviors documented in the current study reflect real-world eye movement strategies. Do the reported inferential anticipatory and reverse saccades generalize to the real-word or are they a consequence of the timing of our study or the particular tasks in the study? The videos were presented for 1.2 s which should not represent an excessive time pressure to observers when compared to other eye movement studies on faces (Peterson and Eckstein, 2012) or search (Ackermann and Landy, 2013; Eckstein et al., 2015). We evaluated a person search task with explicit instructions to follow the gazer and a second task with no eye movement instructions and observed reverse and anticipatory saccades for both tasks. There is also evidence that for natural tasks such as looking at faces or searching, many of the findings with images in the laboratory do generalize to real word settings (Mack and Eckstein, 2011; Peterson et al., 2016). Still, the generalization to the real world for gaze-following eye movement behaviors needs to be assessed with mobile eye trackers (Land and Hayhoe, 2001).

What might be the brain areas involved in the oculomotor programs for gaze following? There is a large literature relating gaze position to neuronal response properties in the superior temporal sulcus (Oram and Perrett, 1994) and dorsal prefrontal cortex (Lanzilotto et al., 2017). These areas relay information to the attention and gaze network in the parietal and frontal cortex which are responsible for covert attention and eye movements (Pierrot-Deseilligny et al., 2004). Finally, the concurrent programming of saccades has been related to neurons in the Frontal Eye Fields (FEF, Basu and Murthy, 2020). Identifying brain areas that integrate peripheral information to generate predictions of likely gaze goals is an important future goal of research.

There are various limitations of the study. In our study, the target/distractor person always appeared at the gaze goal. Thus, the peripheral presence of a person provided some certainty to observers that such a person would be the gazer’s goal. If we had shown videos in which gazers looked at one of two people or other objects even when a person was present, the larger uncertainty about the gazer goals might delay the execution of observer saccades. Thus, cognitive expectations which have been shown to play an important role in oculomotor control (Kowler, 1989) will influence our findings. Our studies used people as gaze goals which are quite visible in the visual periphery. Gaze goals less visible in the observers' periphery will influence the accuracy and timing of the saccades (van Beers, 2007; Han et al., 2021a). In addition, observers might use prior expectations of what types of targets are often looked at (people vs. the floor) to bias their inferences. There is also literature comparing how the human attention system varies when the gaze goals are objects vs. social entities (Mares et al., 2016). Thus, the social nature of the gaze goal, as well as social variables about the gazer, might also influence the oculomotor dynamics investigated (Dalmaso et al., 2020; Macdonald and Tatler, 2013).

Finally, our study focused on the head movement while a large literature focuses on the influences of the gazer’s eyes (Driver et al., 1999; Friesen et al., 2004; Langton et al., 2000; Mansfield et al., 2003; Ristic et al., 2002). Our study was relevant to gazers situated at a distance from the observers. The mean angle subtended by the heads in our videos (1.47°, std = 0.32°) would match the angle subtended by a real-sized head viewed at a distance of 9.3 m (std = 2.0 m) in real life. At that distance, the eye subtends a mean angle of 0.147° (vertically) providing a poor source of information to infer the gaze goals compared to the head orientation. Future studies should investigate gazers at smaller distances from the observer and assess how dynamic gazer eye and head movements are integrated and their interactions (for static images see Balsdon and Clifford, 2018; Cline, 1967; Langton, 2000; Langton et al., 2004; Otsuka et al., 2014). Similarly, we did not analyze lower body movements. Recent studies have shown the diminished influence of the lower body on the orienting of attention (Han et al., 2021b; Pi et al., 2020).

To conclude, our findings reveal the fine-grained dynamics of eye movements while following gaze and the inferential processes the brain uses to predict gaze goals and rapidly program saccades that anticipate the information provided by the gazer’s head direction. Given that attending to the gaze of others is an integral part of a normal functioning social attention system, our findings might provide new granular analyses of eye movement control to assess groups with social attention deficits for which simpler gaze-following analyses have shown disparate results (Chawarska et al., 2003; Nation and Penny, 2008; Ristic et al., 2005).

All data generated or analyzed during this study are deposited at https://osf.io/g9bzt/.

1. 1. Han NX

   (2022) Open Science Framework

   ID g9bzt. Eye Movement Planning During Dynamic Gaze Following.

   https://osf.io/g9bzt/

1. 1. Ackermann JF
   2. Landy MS

   (2013) Choice of saccade endpoint under risk

   Journal of Vision 13:27.

   https://doi.org/10.1167/13.3.27

   • PubMed
   • Google Scholar
2. 1. Araujo C
   2. Kowler E
   3. Pavel M

   (2001) Eye movements during visual search: the costs of choosing the optimal path

   Vision Research 41:3613–3625.

   https://doi.org/10.1016/s0042-6989(01)00196-1

   • PubMed
   • Google Scholar
3. 1. Azzopardi P
   2. Cowey A

   (1993) Preferential representation of the fovea in the primary visual cortex

   Nature 361:719–721.

   https://doi.org/10.1038/361719a0

   • PubMed
   • Google Scholar
4. 1. Bahill AT
   2. Clark MR
   3. Stark L

   (1975) The main sequence, a tool for studying human eye movements

   Mathematical Biosciences 24:191–204.

   https://doi.org/10.1016/0025-5564(75)90075-9

   • Google Scholar
5. 1. Ballard DH
   2. Hayhoe MM
   3. Pelz JB

   (1995) Memory representations in natural tasks

   Journal of Cognitive Neuroscience 7:66–80.

   https://doi.org/10.1162/jocn.1995.7.1.66

   • PubMed
   • Google Scholar
6. 1. Balsdon T
   2. Clifford CWG

   (2018) Task dependent effects of head orientation on perceived gaze direction

   Frontiers in Psychology 9:2491.

   https://doi.org/10.3389/fpsyg.2018.02491

   • PubMed
   • Google Scholar
7. Book

   1. Baron-Cohen S

   (2001) Mindblindness: An Essay on Autism and Theory of Mind

   MIT Press.

   https://doi.org/10.7551/mitpress/4635.001.0001

   • Google Scholar
8. 1. Basu D
   2. Murthy A

   (2020) Parallel programming of saccades in the macaque frontal eye field: are sequential motor plans coactivated?

   Journal of Neurophysiology 123:107–119.

   https://doi.org/10.1152/jn.00545.2018

   • PubMed
   • Google Scholar
9. 1. Becker W
   2. Jürgens R

   (1979) An analysis of the saccadic system by means of double step stimuli

   Vision Research 19:967–983.

   https://doi.org/10.1016/0042-6989(79)90222-0

   • PubMed
   • Google Scholar
10. 1. Bekerman I
    2. Gottlieb P
    3. Vaiman M

    (2014) Variations in eyeball diameters of the healthy adults

    Journal of Ophthalmology 2014:503645.

    https://doi.org/10.1155/2014/503645

    • PubMed
    • Google Scholar
11. 1. Brooks R
    2. Meltzoff AN

    (2005) The development of gaze following and its relation to language

    Developmental Science 8:535–543.

    https://doi.org/10.1111/j.1467-7687.2005.00445.x

    • PubMed
    • Google Scholar
12. 1. Capozzi F
    2. Ristic J

    (2018) How attention gates social interactions

    Annals of the New York Academy of Sciences.

    https://doi.org/10.1111/nyas.13854

    • PubMed
    • Google Scholar
13. 1. Carpenter M
    2. Nagell K
    3. Tomasello M

    (1998)

    Social cognition, joint attention, and communicative competence from 9 to 15 months of age

    Monographs of the Society for Research in Child Development 63:1–143.

    • PubMed
    • Google Scholar
14. 1. Caspi A
    2. Beutter BR
    3. Eckstein MP

    (2004) The time course of visual information accrual guiding eye movement decisions

    PNAS 101:13086–13090.

    https://doi.org/10.1073/pnas.0305329101

    • PubMed
    • Google Scholar
15. 1. Chawarska K
    2. Klin A
    3. Volkmar F

    (2003) Automatic attention cueing through eye movement in 2-year-old children with autism

    Child Development 74:1108–1122.

    https://doi.org/10.1111/1467-8624.00595

    • PubMed
    • Google Scholar
16. 1. Chen CY
    2. Hoffmann KP
    3. Distler C
    4. Hafed ZM

    (2019) The foveal visual representation of the primate superior colliculus

    Current Biology 29:2109–2119.

    https://doi.org/10.1016/j.cub.2019.05.040

    • PubMed
    • Google Scholar
17. Conference

    1. Chong E
    2. Wang Y
    3. Ruiz N
    4. Rehg JM

    (2020) Detecting Attended Visual Targets in Video

    2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5395–5405.

    https://doi.org/10.1109/CVPR42600.2020.00544

    • Google Scholar
18. 1. Cline MG

    (1967)

    The perception of where a person is looking

    The American Journal of Psychology 80:41–50.

    • PubMed
    • Google Scholar
19. 1. Curcio CA
    2. Sloan KR
    3. Packer O
    4. Hendrickson AE
    5. Kalina RE

    (1987) Distribution of cones in human and monkey retina: individual variability and radial asymmetry

    Science 236:579–582.

    https://doi.org/10.1126/science.3576186

    • PubMed
    • Google Scholar
20. 1. Dalmaso M
    2. Castelli L
    3. Galfano G

    (2020) Social modulators of gaze-mediated orienting of attention: A review

    Psychonomic Bulletin & Review 27:833–855.

    https://doi.org/10.3758/s13423-020-01730-x

    • PubMed
    • Google Scholar
21. 1. Driver J
    2. Davis G
    3. Ricciardelli P
    4. Kidd P
    5. Maxwell E
    6. Baron-Cohen S

    (1999) Gaze perception triggers reflexive visuospatial orienting

    Visual Cognition 6:509–540.

    https://doi.org/10.1080/135062899394920

    • Google Scholar
22. 1. Eckstein MP
    2. Schoonveld W
    3. Zhang S
    4. Mack SC
    5. Akbas E

    (2015) Optimal and human eye movements to clustered low value cues to increase decision rewards during search

    Vision Research 113:137–154.

    https://doi.org/10.1016/j.visres.2015.05.016

    • PubMed
    • Google Scholar
23. 1. Egeth HE
    2. Yantis S

    (1997) Visual attention: control, representation, and time course

    Annual Review of Psychology 48:269–297.

    https://doi.org/10.1146/annurev.psych.48.1.269

    • PubMed
    • Google Scholar
24. 1. Emery NJ

    (2000) The eyes have it: the neuroethology, function and evolution of social gaze

    Neuroscience and Biobehavioral Reviews 24:581–604.

    https://doi.org/10.1016/s0149-7634(00)00025-7

    • PubMed
    • Google Scholar
25. 1. Fooken J
    2. Spering M

    (2020) Eye movements as a readout of sensorimotor decision processes

    Journal of Neurophysiology 123:1439–1447.

    https://doi.org/10.1152/jn.00622.2019

    • PubMed
    • Google Scholar
26. 1. Friesen CK
    2. Ristic J
    3. Kingstone A

    (2004) Attentional effects of counterpredictive gaze and arrow cues

    Journal of Experimental Psychology. Human Perception and Performance 30:319–329.

    https://doi.org/10.1037/0096-1523.30.2.319

    • PubMed
    • Google Scholar
27. 1. Gramfort A
    2. Luessi M
    3. Larson E
    4. Engemann DA
    5. Strohmeier D
    6. Brodbeck C
    7. Goj R
    8. Jas M
    9. Brooks T
    10. Parkkonen L
    11. Hämäläinen M

    (2013) MEG and EEG data analysis with MNE-Python

    Frontiers in Neuroscience 7:267.

    https://doi.org/10.3389/fnins.2013.00267

    • PubMed
    • Google Scholar
28. 1. Gregory SEA

    (2022) Investigating facilitatory versus inhibitory effects of dynamic social and non-social cues on attention in a realistic space

    Psychological Research 86:1578–1590.

    https://doi.org/10.1007/s00426-021-01574-7

    • PubMed
    • Google Scholar
29. 1. Han NX
    2. Chakravarthula PN
    3. Eckstein MP

    (2021a) Peripheral facial features guiding eye movements and reducing fixational variability

    Journal of Vision 21:7.

    https://doi.org/10.1167/jov.21.8.7

    • PubMed
    • Google Scholar
30. Preprint

    1. Han NX
    2. Wang WY
    3. Eckstein MP

    (2021b) Gaze Perception in Humans and CNN-Based Model

    arXiv.

    https://arxiv.org/abs/2104.08447

    • Google Scholar
31. 1. Han NX
    2. Eckstein MP

    (2022) Gaze-cued shifts of attention and microsaccades are sustained for whole bodies but are transient for body parts

    Psychonomic Bulletin & Review 29:1854–1878.

    https://doi.org/10.3758/s13423-022-02087-z

    • PubMed
    • Google Scholar
32. 1. Hayhoe M
    2. Ballard D

    (2005) Eye movements in natural behavior

    Trends in Cognitive Sciences 9:188–194.

    https://doi.org/10.1016/j.tics.2005.02.009

    • PubMed
    • Google Scholar
33. 1. Hayhoe MM

    (2017) Vision and Action

    Annual Review of Vision Science 3:389–413.

    https://doi.org/10.1146/annurev-vision-102016-061437

    • PubMed
    • Google Scholar
34. 1. Hooge IT
    2. Erkelens CJ

    (1999) Peripheral vision and oculomotor control during visual search

    Vision Research 39:1567–1575.

    https://doi.org/10.1016/s0042-6989(98)00213-2

    • PubMed
    • Google Scholar
35. 1. Hoppe D
    2. Rothkopf CA

    (2019) Multi-step planning of eye movements in visual search

    Scientific Reports 9:144.

    https://doi.org/10.1038/s41598-018-37536-0

    • PubMed
    • Google Scholar
36. 1. Inhoff AW
    2. Kim A
    3. Radach R

    (2019) Regressions during Reading

    Vision 3:35.

    https://doi.org/10.3390/vision3030035

    • PubMed
    • Google Scholar
37. 1. Intoy J
    2. Rucci M

    (2020) Finely tuned eye movements enhance visual acuity

    Nature Communications 11:795.

    https://doi.org/10.1038/s41467-020-14616-2

    • PubMed
    • Google Scholar
38. 1. Jonides J

    (1981)

    Voluntary versus automatic control over the mind’s eye’s movement

    Attention and Performance.

    • Google Scholar
39. 1. Kaminski J
    2. Riedel J
    3. Call J
    4. Tomasello M

    (2005) Domestic goats, Capra hircus, follow gaze direction and use social cues in an object choice task

    Animal Behaviour 69:11–18.

    https://doi.org/10.1016/j.anbehav.2004.05.008

    • Google Scholar
40. 1. Kingstone A
    2. Smilek D
    3. Ristic J
    4. Kelland Friesen C
    5. Eastwood JD

    (2003) Attention, researchers! It is time to take a look at the real world

    Current Directions in Psychological Science 12:176–180.

    https://doi.org/10.1111/1467-8721.01255

    • Google Scholar
41. 1. Kleinke CL

    (1986)

    Gaze and eye contact: a research review

    Psychological Bulletin 100:78–100.

    • PubMed
    • Google Scholar
42. 1. Kowler E

    (1989) Cognitive expectations, not habits, control anticipatory smooth oculomotor pursuit

    Vision Research 29:1049–1057.

    https://doi.org/10.1016/0042-6989(89)90052-7

    • PubMed
    • Google Scholar
43. 1. Kowler E

    (2011) Eye movements: the past 25 years

    Vision Research 51:1457–1483.

    https://doi.org/10.1016/j.visres.2010.12.014

    • PubMed
    • Google Scholar
44. 1. Kowler E
    2. Rubinstein JF
    3. Santos EM
    4. Wang J

    (2019) Predictive smooth pursuit eye movements

    Annual Review of Vision Science 5:223–246.

    https://doi.org/10.1146/annurev-vision-091718-014901

    • PubMed
    • Google Scholar
45. 1. Lachat F
    2. Conty L
    3. Hugueville L
    4. George N

    (2012) Gaze cueing effect in a face-to-face situation

    Journal of Nonverbal Behavior 36:177–190.

    https://doi.org/10.1007/s10919-012-0133-x

    • Google Scholar
46. 1. Land MF
    2. Hayhoe M

    (2001) In what ways do eye movements contribute to everyday activities?

    Vision Research 41:3559–3565.

    https://doi.org/10.1016/s0042-6989(01)00102-x

    • PubMed
    • Google Scholar
47. 1. Langton SRH
    2. Bruce V

    (1999) Reflexive visual orienting in response to the social attention of others

    Visual Cognition 6:541–567.

    https://doi.org/10.1080/135062899394939

    • Google Scholar
48. 1. Langton SRH

    (2000) The mutual influence of gaze and head orientation in the analysis of social attention direction

    The Quarterly Journal of Experimental Psychology. A, Human Experimental Psychology 53:825–845.

    https://doi.org/10.1080/713755908

    • PubMed
    • Google Scholar
49. 1. Langton SR
    2. Watt RJ
    3. Bruce I

    (2000) Do the eyes have it? Cues to the direction of social attention

    Trends in Cognitive Sciences 4:50–59.

    https://doi.org/10.1016/s1364-6613(99)01436-9

    • PubMed
    • Google Scholar
50. 1. Langton SRH
    2. Honeyman H
    3. Tessler E

    (2004) The influence of head contour and nose angle on the perception of eye-gaze direction

    Perception & Psychophysics 66:752–771.

    https://doi.org/10.3758/bf03194970

    • PubMed
    • Google Scholar
51. 1. Lanzilotto M
    2. Gerbella M
    3. Perciavalle V
    4. Lucchetti C

    (2017) Neuronal encoding of self and others’ head rotation in the macaque dorsal prefrontal cortex

    Scientific Reports 7:8571.

    https://doi.org/10.1038/s41598-017-08936-5

    • PubMed
    • Google Scholar
52. 1. Laubrock J
    2. Cajar A
    3. Engbert R

    (2013) Control of fixation duration during scene viewing by interaction of foveal and peripheral processing

    Journal of Vision 13:11.

    https://doi.org/10.1167/13.12.11

    • PubMed
    • Google Scholar
53. 1. Lee JH
    2. Hwang Shin SJ
    3. Istook CL

    (2006)

    Analysis of human head shapes in the united states

    International Journal of Human Ecology 7:77–83.

    • Google Scholar
54. 1. Leekam SR
    2. Hunnisett E
    3. Moore C

    (1998)

    Targets and cues: gaze-following in children with autism

    Journal of Child Psychology and Psychiatry, and Allied Disciplines 39:951–962.

    • PubMed
    • Google Scholar
55. 1. Legge GE
    2. Klitz TS
    3. Tjan BS

    (1997) Mr. Chips: an ideal-observer model of reading

    Psychological Review 104:524–553.

    https://doi.org/10.1037/0033-295x.104.3.524

    • PubMed
    • Google Scholar
56. 1. Legge GE
    2. Hooven TA
    3. Klitz TS
    4. Stephen Mansfield JS
    5. Tjan BS

    (2002) Mr. Chips 2002: new insights from an ideal-observer model of reading

    Vision Research 42:2219–2234.

    https://doi.org/10.1016/s0042-6989(02)00131-1

    • PubMed
    • Google Scholar
57. 1. Ludwig CJH
    2. Gilchrist ID
    3. McSorley E
    4. Baddeley RJ

    (2005) The temporal impulse response underlying saccadic decisions

    The Journal of Neuroscience 25:9907–9912.

    https://doi.org/10.1523/JNEUROSCI.2197-05.2005

    • PubMed
    • Google Scholar
58. 1. Ludwig CJH
    2. Davies JR
    3. Eckstein MP

    (2014) Foveal analysis and peripheral selection during active visual sampling

    PNAS 111:E291–E299.

    https://doi.org/10.1073/pnas.1313553111

    • PubMed
    • Google Scholar
59. 1. Macdonald RG
    2. Tatler BW

    (2013) Do as eye say: gaze cueing and language in a real-world social interaction

    Journal of Vision 13:6.

    https://doi.org/10.1167/13.4.6

    • PubMed
    • Google Scholar
60. 1. Mack SC
    2. Eckstein MP

    (2011) Object co-occurrence serves as a contextual cue to guide and facilitate visual search in a natural viewing environment

    Journal of Vision 11:1–16.

    https://doi.org/10.1167/11.9.9

    • PubMed
    • Google Scholar
61. 1. Mansfield E
    2. Farroni T
    3. Johnson M

    (2003) Does gaze perception facilitate overt orienting?

    Visual Cognition 10:7–14.

    https://doi.org/10.1080/713756671

    • Google Scholar
62. 1. Mares I
    2. Smith ML
    3. Johnson MH
    4. Senju A

    (2016) Direct gaze facilitates rapid orienting to faces: Evidence from express saccades and saccadic potentials

    Biological Psychology 121:84–90.

    https://doi.org/10.1016/j.biopsycho.2016.10.003

    • PubMed
    • Google Scholar
63. 1. McKay KT
    2. Grainger SA
    3. Coundouris SP
    4. Skorich DP
    5. Phillips LH
    6. Henry JD

    (2021) Visual attentional orienting by eye gaze: A meta-analytic review of the gaze-cueing effect

    Psychological Bulletin 147:1269–1289.

    https://doi.org/10.1037/bul0000353

    • PubMed
    • Google Scholar
64. 1. Meghanathan RN
    2. Nikolaev AR
    3. van Leeuwen C

    (2019) Refixation patterns reveal memory-encoding strategies in free viewing

    Attention, Perception & Psychophysics 81:2499–2516.

    https://doi.org/10.3758/s13414-019-01735-2

    • PubMed
    • Google Scholar
65. 1. Mennie N
    2. Hayhoe M
    3. Sullivan B

    (2007) Look-ahead fixations: anticipatory eye movements in natural tasks

    Experimental Brain Research 179:427–442.

    https://doi.org/10.1007/s00221-006-0804-0

    • PubMed
    • Google Scholar
66. 1. Morales M
    2. Mundy P
    3. Rojas J

    (1998) Following the direction of gaze and language development in 6-month-olds

    Infant Behavior and Development 21:373–377.

    https://doi.org/10.1016/S0163-6383(98)90014-5

    • Google Scholar
67. 1. Morales M
    2. Mundy P
    3. Delgado CEF
    4. Yale M
    5. Messinger D
    6. Neal R
    7. Schwartz HK

    (2000) Responding to joint attention across the 6- through 24-month age period and early language acquisition

    Journal of Applied Developmental Psychology 21:283–298.

    https://doi.org/10.1016/S0193-3973(99)00040-4

    • Google Scholar
68. 1. Mulckhuyse M
    2. Theeuwes J

    (2010) Unconscious attentional orienting to exogenous cues: A review of the literature

    Acta Psychologica 134:299–309.

    https://doi.org/10.1016/j.actpsy.2010.03.002

    • PubMed
    • Google Scholar
69. 1. Najemnik J
    2. Geisler WS

    (2005) Optimal eye movement strategies in visual search

    Nature 434:387–391.

    https://doi.org/10.1038/nature03390

    • PubMed
    • Google Scholar
70. 1. Nation K
    2. Penny S

    (2008) Sensitivity to eye gaze in autism: Is it normal? Is it automatic? Is it social?

    Development and Psychopathology 20:79–97.

    https://doi.org/10.1017/S0954579408000047

    • PubMed
    • Google Scholar
71. 1. Or CCF
    2. Peterson MF
    3. Eckstein MP

    (2015) Initial eye movements during face identification are optimal and similar across cultures

    Journal of Vision 15:12.

    https://doi.org/10.1167/15.13.12

    • PubMed
    • Google Scholar
72. 1. Oram MW
    2. Perrett DI

    (1994) Responses of Anterior Superior Temporal Polysensory (STPa) Neurons to “Biological Motion” Stimuli

    Journal of Cognitive Neuroscience 6:99–116.

    https://doi.org/10.1162/jocn.1994.6.2.99

    • PubMed
    • Google Scholar
73. 1. Otsuka Y
    2. Mareschal I
    3. Calder AJ
    4. Clifford CWG

    (2014) Dual-route model of the effect of head orientation on perceived gaze direction

    Journal of Experimental Psychology. Human Perception and Performance 40:1425–1439.

    https://doi.org/10.1037/a0036151

    • PubMed
    • Google Scholar
74. 1. Pedregosa F
    2. Varoquaux G
    3. Gramfort A
    4. Michel V
    5. Thirion B
    6. Grisel O
    7. Blondel M
    8. Prettenhofer P
    9. Weiss R
    10. Dubourg V
    11. Vanderplas J
    12. Passos A
    13. Cournapeau D
    14. Brucher M
    15. Perrot M
    16. Duchesnay É

    (2011)

    Scikit-learn: Machine learning in python

    The Journal of Machine Learning Research 12:2825–2830.

    • Google Scholar
75. 1. Pelz JB
    2. Canosa R

    (2001) Oculomotor behavior and perceptual strategies in complex tasks

    Vision Research 41:3587–3596.

    https://doi.org/10.1016/s0042-6989(01)00245-0

    • PubMed
    • Google Scholar
76. 1. Peterson MF
    2. Eckstein MP

    (2012) Looking just below the eyes is optimal across face recognition tasks

    PNAS 109:E3314–E3323.

    https://doi.org/10.1073/pnas.1214269109

    • PubMed
    • Google Scholar
77. 1. Peterson MF
    2. Lin J
    3. Zaun I
    4. Kanwisher N

    (2016) Individual differences in face-looking behavior generalize from the lab to the world

    Journal of Vision 16:12.

    https://doi.org/10.1167/16.7.12

    • PubMed
    • Google Scholar
78. 1. Pi Z
    2. Xu K
    3. Liu C
    4. Yang J

    (2020) Instructor presence in video lectures: Eye gaze matters, but not body orientation

    Computers & Education 144:103713.

    https://doi.org/10.1016/j.compedu.2019.103713

    • Google Scholar
79. 1. Pierrot-Deseilligny C
    2. Milea D
    3. Müri RM

    (2004) Eye movement control by the cerebral cortex

    Current Opinion in Neurology 17:17–25.

    https://doi.org/10.1097/00019052-200402000-00005

    • PubMed
    • Google Scholar
80. 1. Rayner K

    (1998) Eye movements in reading and information processing: 20 years of research

    Psychological Bulletin 124:372–422.

    https://doi.org/10.1037/0033-2909.124.3.372

    • PubMed
    • Google Scholar
81. 1. Renninger LW
    2. Verghese P
    3. Coughlan J

    (2007) Where to look next? Eye movements reduce local uncertainty

    Journal of Vision 7:6.

    https://doi.org/10.1167/7.3.6

    • PubMed
    • Google Scholar
82. 1. Ristic J
    2. Friesen CK
    3. Kingstone A

    (2002) Are eyes special? It depends on how you look at it

    Psychonomic Bulletin & Review 9:507–513.

    https://doi.org/10.3758/bf03196306

    • PubMed
    • Google Scholar
83. 1. Ristic J
    2. Mottron L
    3. Friesen CK
    4. Iarocci G
    5. Burack JA
    6. Kingstone A

    (2005) Eyes are special but not for everyone: the case of autism

    Brain Research. Cognitive Brain Research 24:715–718.

    https://doi.org/10.1016/j.cogbrainres.2005.02.007

    • PubMed
    • Google Scholar
84. 1. Rucci M
    2. Iovin R
    3. Poletti M
    4. Santini F

    (2007) Miniature eye movements enhance fine spatial detail

    Nature 447:851–854.

    https://doi.org/10.1038/nature05866

    • PubMed
    • Google Scholar
85. 1. Senju A
    2. Csibra G

    (2008) Gaze following in human infants depends on communicative signals

    Current Biology 18:668–671.

    https://doi.org/10.1016/j.cub.2008.03.059

    • PubMed
    • Google Scholar
86. 1. Shepherd SV

    (2010) Following gaze: gaze-following behavior as a window into social cognition

    Frontiers in Integrative Neuroscience 4:5.

    https://doi.org/10.3389/fnint.2010.00005

    • PubMed
    • Google Scholar
87. 1. Stewart EEM
    2. Valsecchi M
    3. Schütz AC

    (2020) A review of interactions between peripheral and foveal vision

    Journal of Vision 20:2.

    https://doi.org/10.1167/jov.20.12.2

    • PubMed
    • Google Scholar
88. 1. Sun Z
    2. Yu W
    3. Zhou J
    4. Shen M

    (2017) Perceiving crowd attention: Gaze following in human crowds with conflicting cues

    Attention, Perception & Psychophysics 79:1039–1049.

    https://doi.org/10.3758/s13414-017-1303-z

    • PubMed
    • Google Scholar
89. 1. Vallat R

    (2018) Pingouin: statistics in Python

    Journal of Open Source Software 3:1026.

    https://doi.org/10.21105/joss.01026

    • Google Scholar
90. 1. van Beers RJ

    (2007) The sources of variability in saccadic eye movements

    The Journal of Neuroscience 27:8757–8770.

    https://doi.org/10.1523/JNEUROSCI.2311-07.2007

    • PubMed
    • Google Scholar
91. 1. Wallis LJ
    2. Range F
    3. Müller CA
    4. Serisier S
    5. Huber L
    6. Virányi Z

    (2015) Training for eye contact modulates gaze following in dogs

    Animal Behaviour 106:27–35.

    https://doi.org/10.1016/j.anbehav.2015.04.020

    • PubMed
    • Google Scholar
92. 1. Wang L
    2. Yang X
    3. Shi J
    4. Jiang Y

    (2014) The feet have it: local biological motion cues trigger reflexive attentional orienting in the brain

    NeuroImage 84:217–224.

    https://doi.org/10.1016/j.neuroimage.2013.08.041

    • PubMed
    • Google Scholar
93. 1. Wolf C
    2. Schütz AC

    (2015) Trans-saccadic integration of peripheral and foveal feature information is close to optimal

    Journal of Vision 15:1.

    https://doi.org/10.1167/15.16.1

    • PubMed
    • Google Scholar
94. 1. Wolf C
    2. Belopolsky AV
    3. Lappe M

    (2022) Current foveal inspection and previous peripheral preview influence subsequent eye movement decisions

    iScience 25:104922.

    https://doi.org/10.1016/j.isci.2022.104922

    • PubMed
    • Google Scholar
95. 1. Woodward AL

    (2003) Infants’ developing understanding of the link between looker and object

    Developmental Science 6:297–311.

    https://doi.org/10.1111/1467-7687.00286

    • Google Scholar

Decision letter after peer review:

Thank you for submitting your article "Inferential Eye Movement Control while Following Dynamic Gaze" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Miriam Spering as Reviewing Editor and Joshua Gold as the Senior Editor. The reviewers have opted to remain anonymous.

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

1) Please clarify whether cognitive expectations about the role of the second person in videos might have biased gaze patterns and explain the difference between the present/absent conditions (see Reviewer 1).

2) Discuss the choice of creating "absent videos" by digitally removing the target/distractor person from the video, and how this might have affected saccade latency and amplitude variability (Reviewer 1).

3) Consider additional analyses suggested by Reviewer 1.

4) Please explicitly state what observers were asked to do (Methods and repeat in Results); see Reviewer 3.

5) Add requested literature to the introduction and clarify some terminology throughout, please see below.

Reviewer #1 (Recommendations for the authors):

Discussion: The limitations/features of the stimulus material and their potential impact on the results should be considered in the discussion.

The relative contribution of foveal and peripheral information: The authors did a great job in analyzing the relative contribution of foveal and peripheral information, but there might be even further potential for separation: Foveal information on the head direction of the gazer should be quite informative about the direction of the gaze goal, but less about its distance (at least in a 2D plane) and participants might need to rely on peripheral information on potential gaze goals to determine the correct distance. This might influence the error in the saccade direction and saccade amplitude differently. Information about amplitude might be especially uncertain in absent trials where no or multiple objects are in the line of sight of the gazer. I think it might be informative to separately analyze errors in saccade amplitude and direction because they should be related to different pieces of information.

Further analysis: I was wondering to what extent the necessary precision of information about the gazer's head direction depends (a) on the distance between the gazer and the gaze goal and (b) on the nature of the gaze goal. One could expect that higher precision is necessary when the distance between the gazer and the goal is large. For instance, the authors could test in their data how the distance between the gazer and gaze goal influences the occurrence of reverse saccades and the error of the pre- and post-reverse saccades. I admit that this is a bit tangential, but it would be informative as to whether the eye movement pattern is sensitive to the required information for the task. Higher precision might also be necessary when the gaze goal is small or when it is composed of meaningful patterns like a human. For instance, a 10 cm difference in gaze location doesn't matter much when someone is looking at a tree, but it is highly relevant to determine if someone is looking another person in the eyes or not. This aspect is clearly out of the scope of the present study but might be interesting for future studies.

Terminology: I stumbled over the term "reverse" saccades. This seems unnecessarily unspecific because the saccades not only reverse the direction but actually return to the previous fixation location on the gazer. Hence, "return" saccades might be more precise.

Line 77: The difference between the central and peripheral visual fields not only concerns the retina and the cortex, but also subcortical areas like the LGN or the SC.

Line 100: How can gaze cueing be a correlate of ASD? Impairments or alterations in gaze cueing can be correlates of ASD.

Line 158: It is not clear how the error of the first fixation is calculated in absent trials. If there is no person at the gaze goal (or no gaze goal at all), it doesn't seem to be meaningful to calculate an error (at least in terms of distance, the direction might still be well defined by the head direction of the gazer).

Line 203: As the authors describe later in the discussion, the programming of upcoming saccades cannot be altered during the saccadic dead time. The saccadic dead time does not seem to be considered when calculating the gazer vector angular error at the time of the saccade. When one would take the angular error some 50 ms before saccade initiation into account for the dead time, the advantage of prediction would be even larger.

Line 466: Does the fact that the occurrence of reverse saccades depends on the gazer's head velocity imply that humans have some expectation about typical head rotation speeds and assume that a below-average speed indicates the end of the head rotation?

Lines 628 and 644: How well did the annotators agree?

Reviewer #3 (Recommendations for the authors):

I have overall really appreciated this work and I have little to say. My main comments are likely related to some theoretical aspects that could be improved. Indeed, my feelings during my own reading were that this work focuses more on technical aspects and analyses (which is a good point) but less on what is the general meaning that this work provides.

The introduction could be improved a bit. For instance, in lines 98-102, I would also make explicit mentions of some recent reviews on gaze following and gaze cueing (Capozzi and Ristic, 2018; Dalmaso et al., 2020; McKay et al., 2021), to provide naïve readers with a more complete picture about these phenomena.

Then, from line 104: it is true that most studies used static images or faces in isolation, but there are much more exceptions. I am thinking about studies with real social interactions (Lachat et al., 2012; Macdonald and Tatler, 2013) or in multi-agent contexts (e.g., Sun et al., 2017), which could be reported for completeness.

As for theory, if I am right the task required to produce a saccade and then to identify whether a person was the correct target or just a distractor; in some cases, no person at all was presented (except the gazer). What I am missing here is a further condition in which the gazer looks towards a non-social target, such as an object. I am wondering if a different pattern of results could be expected when a target is an object and not another human, as we know that faces and people, when used as targets, can shape eye movements peculiarly s compared to non-social stimuli (Mares et al., 2016). Perhaps I'm missing something obvious here.

References

Capozzi, F., and Ristic, J. (2018). How attention gates social interactions. Annals of the New York Academy of Sciences, 1426(1), 179-198. https://doi.org/10.1111/nyas.13854

Dalmaso, M., Castelli, L., and Galfano, G. (2020). Social modulators of gaze-mediated orienting of attention: A review. Psychonomic Bulletin and Review, 27(5), 833-855. https://doi.org/10.3758/s13423-020-01730-x

Lachat, F., Conty, L., Hugueville, L., and George, N. (2012). Gaze Cueing Effect in a Face-to-Face Situation. Journal of Nonverbal Behavior. https://doi.org/10.1007/s10919-012-0133-x

Macdonald, R. G. R., and Tatler, B. B. W. (2013). Do as eye say: Gaze cueing and language in a real-world social interaction. Journal of Vision, 13(4), 1-12. https://doi.org/10.1167/13.4.6

Mares, I., Smith, M. L., Johnson, M. H., and Senju, A. (2016). Direct gaze facilitates rapid orienting to faces: Evidence from express saccades and saccadic potentials. Biological Psychology, 121, 84-90. https://doi.org/10.1016/j.biopsycho.2016.10.003

McKay, K. T., Grainger, S. A., Coundouris, S. P., Skorich, D. P., Phillips, L. H., and

Henry, J. D. (2021). Visual attentional orienting by eye gaze: A meta-analytic review of the gaze-cueing effect. Psychological Bulletin, 147(12), 1269-1289. https://doi.org/10.1037/bul0000353

Sun, Z., Yu, W., Zhou, J., and Shen, M. (2017). Perceiving crowd attention: Gaze following in human crowds with conflicting cues. Attention, Perception, and Psychophysics, 1-11. https://doi.org/10.3758/s13414-017-1303-z

[Editors’ note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Inferential Eye Movement Control while Following Dynamic Gaze" for further consideration by eLife. Your revised article has been evaluated by Joshua Gold (Senior Editor) and Miriam Spering (Reviewing Editor).

The manuscript has been improved and is almost ready to be accepted, pending just one remaining issue regarding your angular error analysis that needs to be addressed, as outlined in the reviewer's comments below:

Reviewer #1 (Recommendations for the authors):

The authors considerably improved the manuscript and I just have a few remaining comments:

Analysis of saccade errors: I commend the authors for including the angular error of saccades, but I am bit puzzled why they kept the Euclidian distance as the second error metric. The Euclidian distance includes the angular error and therefore the two measures are not completely orthogonal to each other. I would have analyzed the amplitude error in addition to the angular error to have two completely orthogonal measures. Looking at the heat maps in Figure 2, it seems that the deletion of the target/distractor predominantly leads to more elongated landing distributions along the horizontal (gaze) direction. The current way of reporting saccade errors doesn't seem to reflect this increase in anisotropy.

Terminology: I understand that the authors prefer "reverse" over "return" saccades because it matches better their exploration of the data and the narrative in the manuscript. My perspective, however, was about how other researchers will refer to that effect in future studies and in that sense a more precise terminology might be better for future referencing. Ultimately, it's the authors' decision which perspective is more important to them.

Saccadic dead time: estimations of saccadic dead time can vary quite a bit and therefore my suggestion was to use a very conservative estimate at the lower end of the range to avoid exaggerating the effect. However, I'm fine with the author's choice of a more average value of 100 ms.

Essential revisions:

1) Please clarify whether cognitive expectations about the role of the second person in videos might have biased gaze patterns and explain the difference between the present/absent conditions (see Reviewer 1).

We have now added a paragraph (page 24-25) addressing how cognitive expectations might influence the current results. We also discuss other limitations such as how our measurements might vary with the visibility of the gaze goal. In addition, one reviewer also brought up the interesting question of whether the observers’ gaze following behavior might vary for social targets (other people) vs. for non-social inanimate objects. We have also mentioned this as an interesting future possible investigation on page 25.

2) Discuss the choice of creating "absent videos" by digitally removing the target/distractor person from the video, and how this might have affected saccade latency and amplitude variability (Reviewer 1).

We now spell out our motivations for this manipulation, how it relates to more ecologically valid situations and its effects on saccades (page 4). There is an entire paragraph explaining the rationale. We also describe the rationale below in our detailed response to the reviewers.

3) Consider additional analyses suggested by Reviewer 1.

We now incorporated the suggested analyses by Reviewer 1 including the angular error (see new Figures 2d, Figure 5e, and text references throughout the Results section).

4) Please explicitly state what observers were asked to do (Methods and repeat in Results); see Reviewer 3.

We have expanded the description of what the observers were asked to do in the Methods (page 26), the Results section (see page 5). We have also edited the timeline in Figure 1a and the accompanying caption.

5) Add requested literature to the introduction and clarify some terminology throughout, please see below.

We have incorporated the suggested literature in the introduction and discussion.

Reviewer #1 (Recommendations for the authors):

Discussion: The limitations/features of the stimulus material and their potential impact on the results should be considered in the discussion.

The relative contribution of foveal and peripheral information: The authors did a great job in analyzing the relative contribution of foveal and peripheral information, but there might be even further potential for separation: Foveal information on the head direction of the gazer should be quite informative about the direction of the gaze goal, but less about its distance (at least in a 2D plane) and participants might need to rely on peripheral information on potential gaze goals to determine the correct distance. This might influence the error in the saccade direction and saccade amplitude differently. Information about amplitude might be especially uncertain in absent trials where no or multiple objects are in the line of sight of the gazer. I think it might be informative to separately analyze errors in saccade amplitude and direction because they should be related to different pieces of information.

This is a very interesting and insightful point. It would make sense that the peripheral information informed the saccade distance more than the saccade angular error. To investigate the issue, we have now added saccade angular error to all our analyses. These analyses include angular error of first saccades with and without the gaze goal (digital deleting of the gaze goal; Figure 2d); angular error of forward saccade pre and post-reverse saccade (Figure 6b), angular error of forward saccade with gazer removed vs. not removed (Figure 6c).

Here is the interesting and even perhaps surprising result. Our findings show consistency across both measures. The peripheral information about the gaze goal benefited in reducing both the saccade Euclidean error and the saccade angular error. The explanation might lie in the anticipatory nature of the observers’ saccades. The gazer’s head direction is typically pointing about 22 to 28 degrees off from the gaze goal when the observer makes the saccade. Thus, the observer is still needing to make an inference of where the head will point to at the end of the head movement when programming the saccade. Eliminating the peripheral visual information about a potential gaze goal results in an increased error about the gazer’s final head direction.

The rationale (as pointed out by the reviewer!) for a different result for the saccade endpoint Euclidean error vs. the saccade angular error is now explained on page 5. The possible explanation for the results is now discussed on page 21.

Thanks for suggesting the analysis.

Further analysis: I was wondering to what extent the necessary precision of information about the gazer's head direction depends (a) on the distance between the gazer and the gaze goal and (b) on the nature of the gaze goal. One could expect that higher precision is necessary when the distance between the gazer and the goal is large. For instance, the authors could test in their data how the distance between the gazer and gaze goal influences the occurrence of reverse saccades and the error of the pre- and post-reverse saccades. I admit that this is a bit tangential, but it would be informative as to whether the eye movement pattern is sensitive to the required information for the task. Higher precision might also be necessary when the gaze goal is small or when it is composed of meaningful patterns like a human. For instance, a 10 cm difference in gaze location doesn't matter much when someone is looking at a tree, but it is highly relevant to determine if someone is looking another person in the eyes or not. This aspect is clearly out of the scope of the present study but might be interesting for future studies.

We appreciate the thoughtful comments about the role of the eccentricity of the gaze goal, its size and visibility in the periphery. We agree that these are really interesting questions. How the studied effects (saccade error, reverse saccades) vary with systematic changes in gaze goal object size and task requirements (fixating a larger tree vs. a person’s face) is an interesting question. We believe that the real-world video dataset might not be ideal for answering such questions because they do not manipulate each of these variables while controlling for the other.

However, we did analyze the role of eccentricity of the gaze goal in detail which we deemed could be studied to some extent in our data. Here are some of the results below.

– Distance between gazer and gaze goal (eccentricity of gaze goal): We computed the distance between the gazer goal to the gazer measured in terms of degrees and relative to the fovea of the observer (fixating on the gazer). Author response image 1 is the histogram of these eccentricities. For the subsequent analyses, we did a median split of gaze goal eccentricities: median = 4.93° visual angle; low eccentricity (<4.93°) vs. high eccentricity (=4.93°).

Author response image 1

Download asset Open asset

– Saccade properties for low vs. high eccentricity gaze goals: As expected the observer saccade amplitudes to high eccentricity gaze goals were larger than for low eccentricity gaze goals (=4.93° for high eccentricity vs. <4.93° for low eccentricity). There was no difference in the 1st saccade latency with eccentricity.

Author response image 2

Download asset Open asset

– Saccade distance error and angular error vs. gaze goal eccentricity: We computed the error vs. gaze goal eccentricity for experiment 1 and found that there was a similar angular error for the forward saccade of gaze goals at low and high eccentricities (low 14.5 deg vs. high 11.5 deg, p = 0.28 ), but a significantly higher saccade Euclidean error for gaze goals with high eccentricity (low 1.23° vs. high 2.21°, p < 1e-05). Author response image 3 shows the effect of eccentricity.

Author response image 3

Download asset Open asset

– Interaction of eccentricity with the influence of reverse saccades on subsequent fixation error (pre- vs. post-reverse saccade errors): We analyzed how gaze goal eccentricity interacted with the benefit of reverse saccades on subsequent forward saccade errors. We found that both the saccade Euclidean error and the angular error were reduced after the reverse saccade for high eccentricity gaze goals. This seems like an intuitive result. High eccentricity gaze goals have more ambiguity as there are more intervening potential objects between the gazer and the actual gaze goal.Although the various findings are interesting, right now, we did not include these analyses in the revised version of the paper. We felt that investigating these issues required more controlled stimuli that isolated the influence of eccentricity. As an example, the high and low eccentricity gaze goal videos contain different gazer head movements with different head amplitude movements and head velocities. It is difficult to confidently conclude that the differences in observers’ eye movements behavior are solely related to the gaze goal eccentricity. The ideal stimuli to test the influence of gaze-goal eccentricity would be to digitally insert targets along the gaze vectors for the same videos. That would keep the gazer head information the same but vary the gaze goal eccentricity of the videos Right now this is not the case. In addition, we felt that the paper has so many results that adding this would be distracting.

Of course, if the reviewer feels different and wants to see these analyses in the paper, we will be happy to include the results in the supplementary material!

Terminology: I stumbled over the term "reverse" saccades. This seems unnecessarily unspecific because the saccades not only reverse the direction but actually return to the previous fixation location on the gazer. Hence, "return" saccades might be more precise.

Indeed, the reverse saccades do return to the gazer’s head and the term “return” saccades might be more precise. Here is why we use “reverse” saccades.

The paper's narrative follows that of our scientific investigation with the data set. Our analyses found these surprising saccades in the opposite direction but we did not know where these were directed to. Further analyses revealed that observers were re-fixating the gazer’s head. The paper follows that narrative. It identifies the reverse saccades, analyzes their timing and then in a subsequent section investigates where they are directed to. If we called them “return” saccades from the set it would anticipate the result we are uncovering in the section titled: “Functional role of reverse saccades”

That is our rationale to call them “reverse” saccades in the paper. Our preference is to keep it with this narrative. What we have done is insert a sentence in the Discussion section (page 22 line 543) that the reverse saccades could be more precisely described as return saccades. This is inserted in the section (“Functional role of reverse saccades”) after showing that the saccades are re-fixating the gazer’s head. If the reviewer feels a strong preference for changing everything to return saccades we will do so.

Line 77: The difference between the central and peripheral visual fields not only concerns the retina and the cortex, but also subcortical areas like the LGN or the SC.

We have added this information and references to LGN and SC over-representation of the foveal region. Thank you.

Line 100: How can gaze cueing be a correlate of ASD? Impairments or alterations in gaze cueing can be correlates of ASD.

It cannot. Thanks for the correction. We have inserted the word “impairments” as suggested by the reviewer.

Line 158: It is not clear how the error of the first fixation is calculated in absent trials. If there is no person at the gaze goal (or no gaze goal at all), it doesn't seem to be meaningful to calculate an error (at least in terms of distance, the direction might still be well defined by the head direction of the gazer).

It might seem counterintuitive but there is a “gold standard” or “truth” about where the gazer is looking at. It is correct that there is some ambiguity in the images. Even when the gaze goal is present, the person might be looking at a small object that is difficult for the observer to detect peripherally.

We eliminated the gaze goal, so as the reviewer correctly points out, there will be increased ambiguity about where the gazer is looking at. The observer will not have the peripheral visual signal to guide the eye movement and will have to more heavily on the head direction of the gazer.

Line 203: As the authors describe later in the discussion, the programming of upcoming saccades cannot be altered during the saccadic dead time. The saccadic dead time does not seem to be considered when calculating the gazer vector angular error at the time of the saccade. When one would take the angular error some 50 ms before saccade initiation into account for the dead time, the advantage of prediction would be even larger.

Thoughtful comment. We did consider this. But, in the end, we preferred to be conservative and did not add the dead time which would amplify the advantage of saccade prediction. Arguably, having done so would favor our hypothesis. Thus, we felt that including the dead time into our estimation of anticipatory would be perceived by reviewers/readers as convenient to amplify the claim of “anticipatory/inferential eye movements”. Our estimate of the dead time from our studies based on reverse correlation (Caspi et al., 2004) is a little longer than what the reviewer mentions 100 ms. We now mention the calculation with a 100 ms dead time in the Discussion section. If the reviewer feels the evidence for 50 ms is stronger (please provide a reference if this is the case) we will change our numbers.

We are hesitant to include the estimates with the dead time in the abstract and do not have room to explain the nuisance of the two estimates. But if the reviewer feels strongly about this, we will.

Line 466: Does the fact that the occurrence of reverse saccades depends on the gazer's head velocity imply that humans have some expectation about typical head rotation speeds and assume that a below-average speed indicates the end of the head rotation?

Yes, this is what we are suggesting. The analysis of typical gazer head rotations speeds 200 ms before the head stops are similar to the average speed before the reverse saccades (Figure 3c). We now emphasize the reviewer’s point in the discussion so it comes across more clearly (page 22).

Lines 628 and 644: How well did the annotators agree?

We have incorporated into the supplementary material some basic analyses about the annotators’ agreement. The average (across videos) of the standard deviations of the estimation of the gaze goal locations across the ten annotators were: 3.4° horizontally and 1.2° vertically. The average standard deviation across estimated gaze goal angles (between the gazer’s head and estimated gaze goal) of the annotators was: 11.6 deg.

Supplementary Figure S5. includes the full distribution (across videos) of standard deviations (across annotators) of location estimations of the gaze goals. We also included the full distribution of standard deviations of estimated angles. See below for reference.

Reviewer #3 (Recommendations for the authors):

I have overall really appreciated this work and I have little to say. My main comments are likely related to some theoretical aspects that could be improved. Indeed, my feelings during my own reading were that this work focuses more on technical aspects and analyses (which is a good point) but less on what is the general meaning that this work provides.

The introduction could be improved a bit. For instance, in lines 98-102, I would also make explicit mentions of some recent reviews on gaze following and gaze cueing (Capozzi and Ristic, 2018; Dalmaso et al., 2020; McKay et al., 2021), to provide naïve readers with a more complete picture about these phenomena.

We appreciate the reviewer mentioning these. We admit to being new to this subfield and we try to be comprehensive with our literature review. But, we clearly have missed relevant literature. We have now incorporated these references. Thanks for the pointers.

Then, from line 104: it is true that most studies used static images or faces in isolation, but there are much more exceptions. I am thinking about studies with real social interactions (Lachat et al., 2012; Macdonald and Tatler, 2013) or in multi-agent contexts (e.g., Sun et al., 2017), which could be reported for completeness.

Thanks. Appreciated. We have incorporated the reviewer’s suggestions.

As for theory, if I am right the task required to produce a saccade and then to identify whether a person was the correct target or just a distractor; in some cases, no person at all was presented (except the gazer). What I am missing here is a further condition in which the gazer looks towards a non-social target, such as an object. I am wondering if a different pattern of results could be expected when a target is an object and not another human, as we know that faces and people, when used as targets, can shape eye movements peculiarly s compared to non-social stimuli (Mares et al., 2016). Perhaps I'm missing something obvious here.

This is another good point. The question of how the dynamics of the gaze-following eye movements vary when there is a person vs. a non-social target is very interesting. We do not have the data to answer this question but it is certainly and interesting question. Such comparison would have to be implemented by control of the visibility of the social and non-social gaze goals to eliminate low-level confounds. We have included a statement in the discussion (page 24) related to these interesting remaining questions.

[Editors’ note: what follows is the authors’ response to the second round of review.]

The manuscript has been improved and is almost ready to be accepted, pending just one remaining issue regarding your angular error analysis that needs to be addressed, as outlined in the reviewer's comments below:

Reviewer #1 (Recommendations for the authors):

The authors considerably improved the manuscript and I just have a few remaining comments:

Analysis of saccade errors: I commend the authors for including the angular error of saccades, but I am bit puzzled why they kept the Euclidian distance as the second error metric. The Euclidian distance includes the angular error and therefore the two measures are not completely orthogonal to each other. I would have analyzed the amplitude error in addition to the angular error to have two completely orthogonal measures. Looking at the heat maps in Figure 2, it seems that the deletion of the target/distractor predominantly leads to more elongated landing distributions along the horizontal (gaze) direction. The current way of reporting saccade errors doesn't seem to reflect this increase in anisotropy.

Thanks for the thoughtful comment and suggestion. We have added the new figures and analyses on saccade amplitude error to the manuscript (new figure 2,5,6) so that we have a comprehensive understanding of different dimensions of the error measurements (angular error and amplitude error). In addition, we chose to keep the Euclidean error (which combines both angular and amplitude error) because it is easy to interpret. In Figure 2 we show a where we now have all three errors.

Based on the analyses, the saccade amplitude error (absolute difference between saccade amplitude and the distance between the gazer and gaze goal location) is very similar to Euclidean error (distance between the saccade endpoint to the gaze goal location).

Terminology: I understand that the authors prefer "reverse" over "return" saccades because it matches better their exploration of the data and the narrative in the manuscript. My perspective, however, was about how other researchers will refer to that effect in future studies and in that sense a more precise terminology might be better for future referencing. Ultimately, it's the authors' decision which perspective is more important to them.

After some thought, for the sake of the narrative of the paper, we would like to keep the word “reverse saccade” up to the point where we identified that these “reverse saccades” were aimed at the gazer’s heads. We bring up the concept of referring to “reverse saccades” as “return saccades” in the discussion and it might be something we might adopt moving forward.

Saccadic dead time: estimations of saccadic dead time can vary quite a bit and therefore my suggestion was to use a very conservative estimate at the lower end of the range to avoid exaggerating the effect. However, I'm fine with the author's choice of a more average value of 100 ms.

Thanks for the feedback!

Author details

1. Nicole Xiao Han

   Department of Psychological and Brain Sciences, Institute for Collaborative Biotechnologies, University of California, Santa Barbara, Santa Barbara, United States

   Contribution

   Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing – original draft, Project administration, Writing – review and editing

   For correspondence

   xhan01@ucsb.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2860-2743

2. Miguel Patricio Eckstein

   Department of Psychological and Brain Sciences, Department of Electrical and Computer Engineering, Department of Computer Science, Institute for Collaborative Biotechnologies, University of California, Santa Barbara, Santa Barbara, United States

   Contribution

   Conceptualization, Resources, Supervision, Funding acquisition, Validation, Methodology, Writing – original draft, Project administration, Writing – review and editing

   Competing interests

   No competing interests declared

• 152

  Page views

• 27

  Downloads

• 0

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.