Introduction

How the brain represents time remains mysterious. In prospective timing, a starting point is defined beforehand, and the neural activity tracks the elapsed time like a stopwatch. This neural stopwatch is manifested in various forms, including ramping activity, sequential activity, and neural trajectories. (e.g., see reviews by Buonomano & Laje, 2010; Wittmann, 2013; Eichenbaum, 2014; Tsao et al., 2022). However, a stopwatch can only track duration in real time. How can we escape from the present, being able to remember the past or imagine the future?

One solution, which might be unique to humans, is to conceptualize time in terms of space (i.e., the spatial construals of time, e.g., Whorf, 1941; Traugott, 1978; see reviews by Núñez & Cooperrider, 2013; Sinha & Gärdenfors, 2014). This is achieved by segmenting time into events—the basic temporal entities the observer conceives to have a beginning and an end (Zacks & Tversky, 2001)—and ordering these temporal entities in space so that events occurring at different times can be easily maintained in the working memory (Abrahamse et al., 2014; Figure 1A). This reconstructive process consists of two core time concepts: duration (or interval) and sequence (or order).

Figure 1.

Unlike prospective timing tracking the continuous passage of time, durations in time construals are event-based (Tsao et al., 2022): the intervals’ boundaries are constituted by events (Sinha & Gärdenfors, 2014), and the duration of events reflect their span (Núñez & Cooperrider, 2013; Figure 1A). Neurocognitive evidence suggests that prospective and retrospective timing involve separate brain systems. The motor system, especially the supplementary motor area, serves for prospective timing (e.g., Protopapa et al., 2019; Nani et al., 2019; De Kock et al., 2021; Robbe, 2023), whereas the hippocampus contributes to the retrospective one, especially when the duration is embedded within the event sequence (e.g., Barnett et al., 2014; Thavabalasingam et al., 2018; see a comprehensive review by Lee et al., 2020).

As for event sequence, one century ago, the philosopher John McTaggart proposed the distinction between two core time construals: the A-series and the B-series (McTaggart, 1908). The A-series assumes a deictic center of time—the observer’s subjective now—as the reference, and orders events as being in the past or the future according to this deictic center. The B-series concerns the order of events in a sequence regardless of the observer’s subjective now. The distinction between A- and B-series has been widely echoed in various accounts of the temporal frames of reference under different names in the cognitive linguistic analyses (see reviews by Bender & Beller, 2014). Roughly speaking, the A-series corresponds to the egocentric reference frame, whereas the B-series corresponds to the allocentric one.

These two complementary reference frames can be constructed and perceived from two perspectives: one internal, the other external (Núñez & Cooperrider, 2013; Tversky & Jamalian, 2021; Figure 1A). An internal perspective on time series is akin to the “route” perspective in the spatial domain (Siegel & White, 1975). It relates to the cognitive process called “mental time travel” (Tulving, 1984, 2002; Suddendorf et al., 2009). The traveler can project themself into any event in the timeline and redefine past and future according to their self-location (i.e., the travel’s subjective now). In this sense, the brain is no longer a stopwatch but a time machine, taking the traveler back and forth in time (Buonomano, 2017). In contrast, an external viewpoint on time series is akin to the “survey” perspective in the spatial domain (Siegel & White, 1975). It relates to the cognitive process called “mental time watching” (Stocker, 2011, 2014). The watcher can have a panoramic view of multiple events at different times and localize them according to external self-position or temporal landmarks in parallel (e.g., sunrise and sunset in a day or historical events in the long term). In this sense, the brain is more like a dimensional ascension device, taking the watcher out of the one-dimensional timeline to an external viewpoint in higher-dimensional space.

The correspondence between reference frames and perspective-taking is not as straightforward as it seems. In the spatial domain, the conventional assumption suggests that the egocentric reference frame is constructed from the internal route perspective, whereas the allocentric one is from the external survey perspective (e.g., O’Keefe & Nadel, 1978; Arzy & Schacter, 2019). However, as many scholars have pointed out, even from a survey perspective (e.g., when looking at a map), we still have egocentric reference frames (e.g., up, down, left, and right); an allocentric perspective does not exist (e.g., Wolbers & Wiener, 2014; Ekstrom et al., 2014; Filimon, 2015). Going back to the definition, the fundamental distinction between allocentric and egocentric reference frames should lie in whether they change as a function of the perspective taken (Klatzky, 1998): The egocentric representation varies according to perspectives, whereas the allocentric representation remains the same regardless of the perspectives taken. In this vein, the A series can be constructed from both internal and external perspectives as long as there is a fixed self-position as the deictic center (Núñez & Cooperrider, 2013). The B series, per se, is independent of the perspective taken, although an external perspective is usually adopted to construct the B series of time (Núñez & Cooperrider, 2013).

Recent studies have already begun to investigate the neural representation of the memorized event sequence (e.g., Deuker et al., 2016; Thavabalasingam et al., 2018; Bellmund et al., 2019, 2022; see reviews by Cohn-Sheehy & Ranganath, 2017; Bellmund et al., 2020). However, the brain bases underlying the two distinct construals of event sequence remain untouched. Some insights can be drawn from research in the spatial domain. According to an influential neurocomputational model (Byrne et al., 2007; Bicanski & Burgess, 2018; Bicanski & Burgess, 2020), allocentric and egocentric spatial representations are dissociable in the brain—they are respectively implemented in the medial temporal lobe (MTL) and the parietal cortex. Various egocentric representations in the parietal cortex derived from different viewpoints could be transformed into one common allocentric representation and stored in the MTL (i.e., bottom-up process). In return, the allocentric representation in the MTL offers a template from which diverse egocentric representations can be generated in the parietal cortex (i.e., top-down process). Several authors have recently suggested that such mutually engaged egocentric and allocentric reference frames (in the parietal cortex and the medial temporal lobe, respectively) proposed in the spatial domain might also apply to the temporal one (e.g., Gauthier & van Wassenhove, 2016ab; Gauthier et al., 2019, 2020; Bottini & Doeller, 2020). Nonetheless, there is still a lack of neural evidence indicating the parallel involvement of the hippocampal-parietal network in temporal reference frames.

This functional magnetic resonance imaging (fMRI) study aimed to directly test this hypothesis by systematically investigating the neural mechanisms underlying the time construals of event sequence and duration. The event series was a fictional religious ritual of 15 events that participants learned the day before scanning (Figure 1B). Such a ritual had all the core temporal elements: the constituent events followed a specific sequence, endured particular durations, and happened on predetermined parts of the day (i.e., the external temporal landmarks). Participants learned these core temporal elements by reading the description and imagining going through the events one after another. The post-learning test suggests that all participants learned the temporal structure of the ritual before scanning (see Methods for details of the learning and testing procedure).

During the fMRI scanning, participants performed two tasks guiding them to consider the event series from internal and external perspectives (i.e., mental time travel vs. mental time watching; Figure 1C). In each trial, participants saw two sequential event phrases. The first was the reference event, and the second was the target event. In the external-perspective task, participants judged whether the target event happened in the same or different part of the day as the reference event. In the internal-perspective task, participants were instructed to project themselves into the reference event and judge whether the target event happened in the future or the past of the reference event (see Methods for details of the scanning procedure). We compared the neural correlates of event sequence and duration from external and internal perspectives. The egocentric representation of the event series was supposed to vary across the two perspectives, while the allocentric representation was supposed to remain the same.

Results

Time was processed differently from internal and external perspectives

Participants had significantly greater accuracy in the external-perspective task than the internal-perspective task (external-perspective task: M = 93.5%, SD = 4.7%; internal-perspective task: M = 89.5%, SD = 8.1%; paired t(31) = 3.33, p = 0.002). The reaction time (RT) of the corrected trials in the external-perspective task was also significantly shorter than the internal-perspective task (external-perspective task: M = 1475 ms, SD = 529 ms; internal-perspective task: M = 1578 ms, SD = 587 ms; fixed effect of Task Type in a random-intercept-and-slope linear mixed model with Participant as the random-effect grouping factor: F(1, 31) = 27.44, p < 0.001).

To further explore the factors affecting the RT of the correct trials, we built a random-intercept linear mixed model with Participant as the random effects grouping factor. Fixed effects variables included Sequential Distance (i.e., number of events between the reference and the target events), Duration (i.e., duration of the target events), Syllable Length (i.e., number of syllables of the phrase of the target events), Task Type (i.e., external- vs. internal-perspective tasks), and the interaction between Task Type and all the other variables (Table 1). As a sanity check, we found a significant main effect of Syllable Length (F(1, 6918) = 35.59, p < 0.001) since it was expected that participants spent longer time reading longer phrases. Intriguingly, we also found significant interaction effects between Task Type and Sequential Distance (F(1, 6918) = 28.22, p < 0.001) and Task Type and Duration (F(1, 6918) = 12.81, p < 0.001).

Table 1.

Sequential Distance was correlated positively with RT in the external-perspective task (z = 3.80, p < 0.001) but negatively in the internal-perspective task (z = −3.71, p < 0.001). The negative correlation between RT and egocentric distance has been consistently observed in previous studies in which participants engaged in temporal self-projection: participants make more effort to differentiate past and future for the events close to their own temporal position (e.g., Arzy et al., 2008; Arzy et al., 2009ab; Gauthier & Wassenhove, 2016ab; Gauthier et al., 2019). This pattern can broadly be attributed to the symbolic distance (SD) effect (Moyer & Landauer, 1967; Moyer & Bayer, 1976; Shepard & Judd, 1979), which refers to the fact that “time needed to compare two symbols varies inversely with the distance between their referents on the judged dimension” (Moyer & Bayer, 1976, p. 229). The positive correlation between RT and sequential distance from an external perspective was predicted by Gauthier & van Wassenhove (2016a). This prediction was inspired by classic studies on mental scanning using visual imagery (e.g., Shepard & Metzler, 1971; Kosslyn et al., 1978): From an external perspective, people might compare two referents by mentally traversing the intermediate states between them, resulting in longer times to scan longer sequential distances.

As for Duration, it had a significantly negative trend with RT in the external-perspective task (z = −4.44, p < 0.001) but not in the internal-perspective task (z = 0.66, p = 0.51). A possible explanation is that events with longer durations were more salient and thus easier to be compared with external landmarks.

The behavioral results suggest that our attempt to induce different perspectives on the event series was successful. The two tasks induced RT as distinct functions of sequential distance, in line with the predicted SD effect for the internal perspective and reverse-SD effect for the external perspective.

Internal- compared to external-perspective task activated different brain networks

We first directly contrasted the activity level between external- and internal-perspective tasks in the time window of the target events (Figure 2A; Table 2; see Supplementary Figure 1A for a surface view; voxel level p < 0.001, cluster-level FWE corrected p < 0.05). Compared with the external-perspective task, the internal-perspective task specifically activated the regions in the default network (DN) in the right hemisphere. They were the precuneus (PreC), the retrosplenial cortex (RSC), the superior frontal gyrus (SFG), and the angular gyrus (AG). This finding aligned with evidence indicating that the DN plays a crucial role in self-projection (see the review by Buckner & Carroll, 2007). In Supplementary Figure 1, we also compared the significant clusters in the internal-perspective task with the two subnetworks of the DN: DN-A and DN-B (e.g., Braga & Buckner, 2017; Reznik et al., 2013). The internal-perspective mostly engages DN-A rather than DN-B. This observation is consistent with existing evidence suggesting that DN-A is more closely associated with episodic memory, whereas DN-B is primarily involved in social processing (e.g., Lin et al., 2018; DiNicola et al., 2020).

Figure 2.

Table 2.

Compared with the internal-perspective task, the external-perspective task specifically activated the supplementary motor area (SMA) and the supramarginal gyrus (SMG) in the right hemisphere (Figure 2A; Table 2; voxel level p < 0.001, cluster-level FWE corrected p < 0.05). This finding is open to interpretation. The area found in the right SMG was centered at the junction region between the posterior part of the SMG and the posterior part of the superior temporal gyrus. Previous evidence has shown that this temporoparietal junction area relates to the out-of-body experience (e.g., see reviews by Blanke et al., 2004; Blanke & Arzy, 2005) or plays a role in overcoming egocentric emotion bias towards others (e.g., Silani et al., 2013). Thus, the right SMG region found here may be important for the mental construction of an external perspective.

The right posterior parietal cortex implemented opposite representations of sequential distance across external and internal perspectives

Next, we used the parametric modulation method to detect neural correlates of the temporal information (i.e., Sequential Distance and Duration) across external- and internal-perspective tasks. We built a single general linear model (GLM) in which the target events were simultaneously modulated by their duration and their sequential distances to the reference events. The target events in external- and internal-perspective tasks were treated separately. To detect whether the temporal information was represented differently from different perspectives, we first examined the interaction effect between each temporal information and the task type. If the interaction effect was not significant, we also examined the main effect combining both tasks.

As a sanity check, we investigated whether the employed parametric modulation method could successfully reveal the effect of the word length in the visual cortex, given that our stimuli were visually presented. To do so, we used Syllable Length as the parameter to modulate the condition of both the reference and the target events in the above GLM. We found not only the visual cortex but also the left superior temporal gyrus in the language network, of which the activity level positively correlated with the number of syllables (Figure 2B; voxel level p < 0.001, cluster-level FWE corrected p < 0.05). This result confirmed our prediction and validated our methodology.

To detect the regions in which the activation level was modulated by Sequential Distance (i.e., the number of events between the reference and the target events), we first searched regions with a significant interaction effect between Task Type (i.e., external- vs. internal-perspective tasks) and Sequential Distance (Figure 3A-C). The only significant region we could find across the whole cortex was localized in the border area between the angular gyrus and the superior division of the lateral occipital cortex in the right hemisphere (i.e., the boundary area between Broadman area 7, 19, and 39; Figure 3A; voxel level p < 0.001, cluster-level FWE corrected p < 0.05). More specifically, this region is mostly at the lateral wall of the posterior intraparietal sulcus (hIP5: 56.2%, hIP6: 9.5%, hIP4: 5.9%; Richter et al., 2019) and the posterior part of the angular gyrus (PGp: 21.4%; Caspers et al., 2006, 2008) (assignment based on maximum probability map; Eickhoff et al., 2005). The MNI coordinate of the center voxel is 38, −69, 35. For convenience, we will refer to this region as the posterior parietal cortex (PPC) in the following text. Figure 3B further shows the beta estimates of Sequential Distance in the parametric modulation analysis in the right PPC. The activation level in this region correlated with sequential distance positively in the external-perspective task (t(31) = 2.97, P = 0.006) but negatively in the internal-perspective task (t(31) = −4.19, P < 0.001).

Figure 3.

The alignment of the changes in the activity levels in the PPC with RT suggests a potential domain-general task difficulty effect. To eliminate this possibility, we controlled for trial-by-trial RT using the parametric modulation method in our first-level analysis (see Methods). Notably, the same PPC region remained the only area in the entire brain showing a significant interaction effect between Task Type and Sequential Distance (voxel-level p < 0.001, cluster-level FWE corrected p < 0.05). This indicates that the PPC activity cannot be fully attributed to RT.

To provide a direct illustration of how the activation level in the right PPC varied according to the sequential distances in the two perspectives, we built another GLM with each sequential distance (i.e., 1 to 5) in each task as a separate condition. Figure 3C confirms that the activation level in the external-perspective task went up as the sequential distance increased, and the activation level in the internal-perspective task went down as the target event moved far away from the reference event where participants self-projected themselves.

These results suggest that the parietal cortex implements an egocentric representation of the event sequence, which varies with different perspectives.

The right hippocampal head implemented consistent representations of sequential distance across external and internal perspectives

We did not find any regions in the hippocampus showing a significant interaction effect between Task Type and Sequential Distance. Instead, we found a region in the right hippocampal head in which the activation level positively correlated with the sequential distance when we combined external- and internal-perspective tasks (Figure 3D; voxel level p < 0.001, cluster-level FWE corrected p < 0.05). The MNI coordinate of the center voxel was 25, −11, −15. Figure 3E further shows that we did not find that such a positive correlation with the sequential distance significantly differed in the external- and internal-perspective tasks (paired t(31) = −0.906, p = 0.372).

We also illustrate the activation level in this hippocampal region of each sequential distance in external- and internal-perspective tasks, respectively (Figure 3F): the activation level in the right hippocampus tended to go up as the sequential distance increased regardless of the tasks.

These results suggest that the hippocampus implements an allocentric representation of the event sequence independent of the perspectives, contrary to the egocentric representation in the posterior parietal cortex.

The right hippocampal body implemented the representation of event duration in the internal-perspective task

In the cortex, we did not detect any regions where the activation level showed a significant interaction effect between Task Type and Duration or a positive main effect of Duration. In the hippocampus, we also found no regions where the activation level showed a significant interaction effect between Task Type and Duration. Instead, we found a region in the right hippocampal body where the activation level showed a significantly positive correlation with Duration when combining two tasks (Figure 4A; voxel level p < 0.001, cluster-level FWE corrected p < 0.05). The MNI coordinate of the center voxel is 39, −24, −12. However, Figure 4B shows that directly comparing the beta estimates in the two tasks reveals a significant result (paired t(31) = 3.07, p = 0.004). The mean activation level in this area positively correlated with Duration only in the internal-perspective task (t(31) = 5.54, p < 0.001), not in the external-perspective task (t(31) = 0.68, p = 0.502). This later analysis is circular, as the ROI was defined as the voxels showing a significantly positive correlation with Duration combining two tasks in the first place. Nevertheless, it illustrated that this positive correlation in the voxel-level analysis was driven only by the internal-perspective task. Due to the stringent threshold of multiple comparisons across voxels, this interaction effect between Task Type and Duration was not found in the voxel-level analysis.

Figure 4.

Figure 4C directly illustrates how the activation level in this hippocampal region varied according to duration: the activation level in the right hippocampal body went up as the duration increased only in the internal-perspective task but not in the external-perspective task.

A possible explanation for the difference between these two tasks might be that the hippocampus is preferentially involved in memory for durations embedded within event sequences (see review by Lee et al., 2020). Since the external-perspective task in the current study encouraged the participants to compare the event sequence with the external parallel temporal landmarks, duration representation in the hippocampus may be dampened.

We also implemented a post hoc analysis to fully illustrate the Sequential Distance and Duration effects in the internal-perspective task within the regions of interest of the hippocampal Head and the Body (Figure 4D). These two regions were defined as the regions where activation level significantly positively correlates with Sequential Distance and Duration, respectively. No significant evidence shows that the hippocampal head also represented Duration (t(31) = 0.506, p = 0.616), and no significant evidence shows that the hippocampal body also represented Sequential Distance (t(31) = −0.616, p = 0.543). Further confirmatory analyses are needed to verify such a double dissociation pattern between the neural representation of sequence and duration in the head and the body of the hippocampus.

Discussion

This study investigated the neural correlates of sequence and duration in task contexts where participants took internal or external perspectives on event series, thereby testing the complementary allocentric and egocentric reference frames in the temporal domain. We found that both the right PPC and the right hippocampal head represented the sequential distance between the reference and the target events. However, the representation in the right PPC varied with the perspective taken; its activity level correlated with sequential distance positively in the external perspective task and negatively in the internal perspective task. In contrast, the activation level in the right hippocampal head positively correlated with sequential distance regardless of the perspective taken. Moreover, we found that the activation level in the right hippocampal body positively correlated with the event duration in the internal-perspective task.

The negative correlation between the activation level in the right PPC and sequential distance has already been observed in a previous fMRI study (Gauthier & van Wassenhove, 2016b). The authors found a similar region (the reported MNI coordinate of the peak voxel was 42, −70, 40, and the MNI coordinate of the peak voxel in the present study was 39, −70, 35), of which the activation level went up when the target event got closer to the self-positioned event. This finding aligns with the evidence suggesting that the posterior parietal cortex implements egocentric representations. For example, neuropsychological studies have demonstrated that patients with lesions in the PPC have “egocentric disorientation” (Aguirre & D’Esposito, 1999): they are unable to localize objects in relation with themselves (e.g., Case 2: Levine et al., 1985; Patient DW: Stark, 1996; Patient MU: Wilson et al., 1997, 2005). Consistently, we found in a recent fMRI experiment that the distributed activity pattern in the bilateral PPC could encode egocentric but not allocentric coordinates of the target objects during memory retrieval (Dutriaux et al., 2024).

What is novel here is that the correlation between the activation level of the right PPC and sequential distance was reversed to a positive one when an external perspective was used. Since an external perspective is often associated with the allocentric reference frame (e.g., O’Keefe & Nadel, 1978; Arzy & Schacter, 2019), our results seem to challenge the view that the parietal cortex implements egocentric representation. However, the conflict is more apparent than real. Perspective-taking, whether internal or external, by definition, must be egocentric, as mental images must be constructed from a specific viewpoint (see similar arguments by Vogeley & Fink, 2003; Filimon, 2015). Therefore, a brain region that implements egocentric representation should vary its activity with the perspectives taken, as shown in the right PPC. This finding is crucial; it suggests that the distance coding in the right PPC is not a “perspective agnostic” magnitude effect, a possibility that previous fMRI studies could not definitively rule out.

Several previous studies have already quested the nature of the PPC’s function (e.g., see reviews by Andersen et al., 1997; Cohen & Andersen, 2002; Wagner et al., 2005; Abrahamse et al., 2014; Ciaramelli et al., 2008; Cabeza et al., 2008; Hutchinson et al., 2009; Whitlock, 2017; Sestieri et al., 2017; Summerfield et al., 2020; Bottini & Doeller, 2020). Here, we want to highlight three crucial findings. First, neuropsychological studies indicate that lesions in the bilateral PPC or lateral occipitoparietal cortex will lead to not only egocentric disorientation but also simultanagnosia, one of the key components of Bálint’s syndrome (Bálint, 1909; Rizzo & Vecera, 2001; Chechlacz & Humphreys, 2014; e.g., WF: Holmes and Horax, 1919; MVV: Kase et al., 1977; MU: Wilson et al., 1997, 2005). Patients with simultanagnosia cannot perceive multiple entities as a whole and comprehend the overall meaning of a scene. Second, electrophysiological recordings in the lateral intraparietal area of the macaque cortex find the gain-field neurons, which can be used to transform the reference frames anchored to different body parts (e.g., Andersen et al., 1985; Zipser & Andersen, 1988; Andersen et al., 1997; Cohen & Andersen, 2002). Third, the fMRI studies in humans show that the particular PPC area found in our study (i.e., the hIP5 area in the lateral wall of the intraparietal sulcus and the dorsal PGp in the angular gyrus) are more engaged in episodic memory retrieval, in contrast to the medial wall of the intraparietal sulcus and dorsal part of the PPC more involved in perceptual attention (e.g., Hutchinson et al., 2009; Sestieri et al., 2010; Sestieri et al., 2017). Taking all these findings together, the PPC area identified in this study might contribute to memory retrieval in egocentric reference frames, which maintains relations among multiple memorized entities in the working memory from a perspective optimal for the current task context (Abrahamse et al., 2014). Such relational schema, constructed from stereotypical perspectives, is also represented in the PPC as a template to guide attention (Summerfield et al., 2020; Bottini & Doeller, 2020). It is thus not surprising that the two most common perspectives for time construals—mental time travel and mental time watching—involve the PPC.

In contrast to the PPC, we observed that the activation level in the head of the right hippocampus positively correlated with the sequential distance, regardless of the perspectives. Previous studies have already shown that the hippocampal activation level correlates with distance (e.g., Morgan et al., 2011; Howard et al., 2014; Theves et al., 2019, Viganò et al., 2023), and the distributed activity in the hippocampus can encode distance (e.g., Deuker et al., 2016; Park et al., 2021). However, our study is novel in showing that this distance coding in the hippocampus is independent of perspectives, indicating an allocentric representation of the event series. This finding aligns with the hypothesis in the spatial domain suggesting the hippocampus as an allocentric cognitive map (e.g., O’keefe & Nadel, 1978; Byrne et al., 2007; Bicanski & Burgess, 2018; Bicanski & Burgess, 2020; Bottini & Doeller, 2020). The positive correlation between the sequential distance and the hippocampal activation level might be mediated by adaptation (e.g., Grill-Spector & Malach, 2001): the longer the distance, the less the adaptation, and the greater the hippocampal activation. One way to interpret such perspective-agnostic representation in the hippocampus is to view it as the associations among memorized entities (e.g., Muller et al., 1996; Eichenbaum, 2014, 2017; Buzsáki & Tingley, 2018; Quiroga, 2019). As a result, the allocentric representation in the hippocampus may not require a “reference frame” because there is no fixed reference point. Each entity is represented among the relations to others, contrasting with the egocentric representation in the parietal cortex, where the reference point is clearly the self.

In this context, the distinction between the allocentric and the egocentric representations of an event series can also be understood in terms of memory storage and retrieval (also see Byrne et al., 2007). The hippocampus stores event series in a static and perspective-agnostic manner, while the PPC flexibly retrieves memory by constructing egocentric images tied to variable perspectives. Supporting this hypothesis is evidence that bilateral hippocampal lesions result in memory deficit for event series (e.g., Dede et al., 2016), whereas bilateral PPC lesions impair free recall but do not cause memory loss (e.g., Berryhill et al., 2007). Patients can still recall details of past events when prompted with cues or questions but struggle to access memories spontaneously.

Finally, the study indicates that the event durations were represented in the right hippocampal body, in line with previous studies suggesting that the hippocampus contributes to the retrospective durations that are embedded within the event sequence (e.g., Barnett et al., 2014; Thavabalasingam et al., 2018; Lee et al., 2020). Notably, the post hoc analysis reveals a double dissociation pattern: the hippocampal head represented sequential distances between events, whereas the hippocampal body represented the duration of individual events. This novel finding echoes the evidence suggesting that the anterior hippocampus (or the ventral rodent hippocampus) implements global and gist-like representations (e.g., larger receptive fields), whereas the posterior hippocampus (or the dorsal rodent hippocampus) implements local and detailed ones (e.g., finer receptive fields) (e.g., Jung et al., 1994; Kjelstrup et al., 2008; Collin et al., 2015; see reviews by Poppenk et al., 2013; Robin & Moscovitch, 2017; see Strange et al., 2014 for a different opinion). Thus, the hippocampus likely represents the sequence of events hierarchically along its longitudinal axis: from the gist event sequence to the event sequence constituting a gist event, down to unitary time bin sequence constituting an event as its duration. This view unifies the hippocampus’s role in sequence representation and explains why it only represents durations embedded within event sequences. Future studies are required to confirm this hypothesis and investigate whether such hierarchical structures along the longitude axis of the hippocampus can also be used to represent the events at different time scales (e.g., from years to seconds). It would also be intriguing to examine whether the parietal cortex has a similar hierarchical structure or plays a role in zoom-in and zoom-out events at different scales to maintain an appropriate working memory load.

To conclude, this study reveals the neural correlates of sequence and duration in time construals. The hippocampal head represents the event sequence allocentrically, irrespective of the observer’s perspectives (i.e., the B series), whereas the hippocampal body represents the event duration embedded in the event sequence. The posterior parietal cortex flexibly constructs the egocentric representation of the event series that varies according to the observer’s perspectives (i.e., the A series), which could be internal (i.e., mental time travel) or external (i.e., mental time watching). Such allocentric and egocentric representations of event series can be interpreted in terms of memory storage and retrieval.

Methods

Participants

Thirty-five native Italian speakers with no history of neurobiological or psychiatric disorders participated in the experiment. The ethical committee of the University of Trento approved the experimental protocol, and all participants provided written informed consent and were paid for their time. Three participants were excluded. Two were due to poor behavioral performance during scanning; their accuracy was below 1.5 times the interquartile range below the lower quartile across participants. One was due to excessive head motion during scanning; this participant’s mean frame displacement index (Power et al., 2014) was above 1.5 times the interquartile range above the upper quartile across participants. The remaining 32 participants entered the analysis (19 females, 13 males; age: 19-34; M = 24.0; SD = 3.8; all participants except one were right-handed).

Stimuli

We created a fictional religious ritual as the stimulus. Like most rituals, it follows a specific sequence, endures particular durations, and happens on predetermined parts of the day (Figure 1A). This created ritual comprised 15 events, falling into three parts of the day, i.e., morning, afternoon, and evening. Each part of the day included five events and lasted six hours in total: two events lasted for half an hour, one for one hour, and the other two for two hours. We instantiated each event with an event phrase. To minimize the potential confounding between the time information of the events and the semantic information of the phrases, we randomly assigned 15 phrases to the events twice, generating two versions for even and odd numbers of participants.

Procedures

The day before fMRI scanning, participants learned the temporal information of the ritual: the sequence, the durations, and the parts of the day. The learning procedure included two phases. The first was the reading phase, and the second was the imagination phase. The two phases combined were performed twice.

In the reading phase, participants read a narrative describing the whole ritual on a computer screen twice. The computer screen presented one sentence at a time, and each sentence provided information on one event. The presentation sequence was identical to the event sequence of the fictional ritual, and the sentences described the duration and the parts of the day of each event. Participants read the sentences at their own pace by pressing the spacebar to read the following sentence.

In the imagination phase, participants imagined themselves performing the whole ritual one event after another, guided by the prerecorded auditory instructions delivered through the headphones. Each event started with a voice telling the event to be imagined (e.g., “light some candle”). A single beep was then played to indicate the start of the imagination, and a double beep sound indicated the end of the imagination. The imagining sequence was the same as the actual sequence, the imagining duration (i.e., the interval between the single and the double beep) was proportional to the actual duration (30 seconds / 1 hour), and the parts of the day were indicated in the instruction (e.g., “the morning starts”). Participants were told to close their eyes to avoid distractions and imagined the whole ritual four times in each imagining phase (i.e., eight times in total).

After learning, participants’ knowledge of the ritual was assessed with three tests. In the event-sequence test, participants judged whether one event happened in the past or the future of another event. In the event-duration test, participants imagined the ritual in a self-paced manner and pressed the button when finishing the imagination of each event. In the parts-of-the-day test, participants judged whether two events happened in the same or different parts of the day. All participants’ performance was greater than 80% in both the event-sequence and parts-of-the-day tests. In the event-duration test, Pearson’s correlation between the self-paced imagining duration and the actual duration across 15 events was greater than 0.6 in all participants.

The scanning consisted of six runs, each with one external-perspective task block and one internal-perspective task block. The order of these two task blocks was interleaved across the six runs within each participant, and the order of these two task blocks in the first run was counterbalanced across participants. That means half the participants followed the order “EI-IE-EI-IE-EI-IE”, and half of the participants followed the order “IE-EI-IE-EI-IE-EI” (“E” indicating the external-perspective task block and “I” indicating the internal-perspective task block). Each task block started with a 5s task prompt indicating the task of this block and a 3s countdown presentation (i.e., the screen presenting “3”, “2”, “1”). The task block had 20 trials that were identical in the two task blocks within the same run but were presented in a randomized order.

In each trial, the phrases of two events were visually presented one after the other (Figure 1C). Participants first saw a 0.5s fixation cross, the phrase of the reference event for 0.8s, and a blank screen for 3.2s (75% trials) or 7.2s (25% trials). They then saw another 0.5s fixation cross and the phrase of the target event. Each event phrase was presented in the center of a grey screen in two rows (black color, Calibri font): in the first row was the verb of the phrase (i.e., “light”), and in the second row was the object of this verb “some candle”). When presenting the target event phrase, we also presented two possible answers under the target event phrase; they were smaller in font size and colored differently depending on the task (i.e., red for the external-perspective task and blue for the internal-perspective task). In the external-perspective task block, the participants judged whether the target event happened in the same or different part of the day of the reference event, and the two option words were “same” and “different”. In the internal-perspective task block, the participants were instructed to project themselves into the reference event and judge whether the target event happened in the past or will happen in the future, and the two option words were “past” and “future”.

Participants were instructed to perform the two tasks once they saw the target event phrase by pressing two buttons with their right index and middle fingers, corresponding to the left and the right options on the screen, respectively. To ensure that the behavior output did not correlate with the finger movement, we counterbalanced the locations of the two options on the screen across the trials within both external- and internal-perspective blocks. The reaction time was calculated as the duration between the onset of the target event phrase and the button press. A blank screen would replace the target event presentation once the response was given, and the total duration of the target event and the blank screen was maintained as 4s. On 75% of occasions, the subsequent trial started immediately; On the other 25%, there would be another 4s blank screen between trials.

We carefully chose two sets of 20 pairs of events from the possible 210 pairs of events. These two sets were assigned to odd and even runs of the fMRI experiment, respectively. Using the brute-force search method, we searched 20 pairs with weak correlation between the sequential distance and the position information of both the reference event and the target event (i.e., from 1 to 15) (all the correlation coefficients < 0.2). At the same time, we also balanced the corrected choice ratio during both the external-perspective task (Same/Different = 11/9, 12/8) and the internal-perspective task (Future/Past = 12/8, 8/12). Given the criteria, sequential distances in both sets fell within the range between 1 and 5.

Behavior analysis

Behavior analyses were conducted using JASP 0.18.3. To investigate the factors affecting trial-by-trial RT of the correct trials, we built linear mixed models (LMM) with Participant as the random effects grouping factor. We fit the maximal model including the random intercept and all the random slopes consistent with the experimental design (as recommended by Barr et al., 2013). In the case of overfitting (singular fit), we removed the random slopes but kept the random intercept (Matuschek et al., 2017).

MRI acquisition

MRI data were acquired using a MAGNETOM Prisma 3T MR scanner (Siemens) with a 64-channel head-neck coil at the Centre for Mind/Brain Sciences, University of Trento. Functional images were acquired using the simultaneous multislice echoplanar imaging sequence: the scanning plane was parallel to the long axis of the hippocampus, the phase encoding direction was from anterior to posterior, repetition time (TR) = 1000 ms, echo time (TE) = 28 ms, flip angle (FA) = 59°, field of view (FOV) = 200 mm × 200 mm, matrix size = 66 × 66, 65 axial slices, slices thickness (ST) = 3 mm, no gap, voxel size = 3.03 × 3.03 × 3 mm, multiband factor = 5.

Field maps were acquired between each pair of functional runs to correct for geometric distortions of these two runs. Each set of field maps included three images, one phase-drift image between two slightly different echo times and two magnitude images for each of these two echo times: the scanning plane was parallel to the long axis of the hippocampus, the phase encoding direction was from anterior to posterior, TR = 1030 ms, shorter TE = 4.92 ms, longer TE = 7.38 ms, FA = 60°, FOV = 210 mm × 210 mm, matrix size = 70 × 70, 66 axial slices, ST = 3 mm, no gap, voxel size = 3 × 3 × 3 mm.

Three-dimensional T1-weighted images were acquired using the magnetization-prepared rapid gradient-echo sequence, sagittal plane, TR = 2530 ms, TE = 1.69 ms, inversion time = 1100 ms, FA = 7°, FOV = 256 mm × 256 mm, matrix size = 256 × 256, 176 continuous sagittal slices, ST = 1 mm, voxel size = 1 × 1 × 1 mm.

MRI preprocessing

We preprocessed the brain images using SPM12 (https://www.fil.ion.ucl.ac.uk/spm/software/spm12/). The functional images were first realigned to the first image in the first run, generating six rigid head motion parameters for each time point and a mean functional image across all the runs. We then used the field maps to calculate the voxel displacement maps and coregistered them to this mean functional image. Since we acquired the field maps between each pair of functional runs, we applied each voxel displacement map to its two closet functional runs to correct their geometric distortions. The resulting functional images were next normalized to the MNI space with the acquired T1-weighted image using the unified segmentation method. In the final step, we spatially smoothed the normalized functional images, and the full width at half maximum of the 3D Gaussian smoothing kernel was 4 mm.

First-level fMRI analysis

We performed the first-level analysis using SPM12 (https://www.fil.ion.ucl.ac.uk/spm/software/spm12/). General linear models (GLMs) were built to predict each participant’s blood-oxygen-level-dependent (BOLD) signal. The GLMs in the primary analysis included six conditions: the task prompt and the countdown at the beginning of each block, the reference event and the target event in the external-perspective task, and the reference event and the target event in the internal-perspective task. The duration of the task prompt and the countdown were set as the duration of the actual presentation (i.e., 5s and 3s, respectively). The duration of the four event conditions was set as 0. The resulting boxcar and stick functions were convolved with a canonical hemodynamic response function (HRF). Head motion parameters and constant variables indicating each of the six runs were included as nuisance regressors. A high-pass filter with a cutoff of 128s was used to remove the low-frequency noise and slow signal drifts.

We used the parametric modulation method to investigate the two core temporal variables: sequence (i.e., the number of events between the reference event and the target event) and duration (i.e., the duration of the target events in terms of hours). The z-scores of these parameters across the events in each run were set as parameters modulating the target events in both external- and internal-perspective tasks. The option for orthogonalizing modulations in the SPM was turned off (Mumford et al., 2015).

As a sanity check, we investigated whether the parametric modulation method in this study can successfully detect the visual cortex whose activation should be modulated by the number of syllables of the visually presented event phrase. To this end, we built a stick function with the sticks located on the onsets of both reference and target events and the “height” of the stick as the z score of the number of syllables across the event phrases in each run. This stick function was convolved with a canonical HRF as an additional regressor involved in the GLM.

To detect the specific activations for external- and internal-perspective tasks, we directly contrasted the target event in the external-perspective task and the target event in the internal-perspective task. To investigate how the neural activation was modulated by each of the temporal information across different perspectives, we looked at its interaction effect with the task (i.e., contrast the effect between external- and internal-perspective tasks, i.e., contrast weights vector: 1, −1) and the main effect regardless of tasks (i.e., contrast weights vector: 1, 1).

To validate whether any significant effects were explained by the RT, we also built the same GLM incorporating trial-by-trial RT as a covariate. To do so, we created a stick function with the sticks positioned at the onset of the target events in both external- and internal-perspective tasks, and the height of each stick was set to the z-score of the corresponding RT. These sticks were convolved with a canonical HRF to serve as a regressor in the GLM. To directly illustrate how the activation level varied with the sequential distance, we built an independent GLM with each distance (i.e., from 1 to 5) in each task as a separate condition (i.e., ten conditions in total).

To directly illustrate how the activation level varied with duration, we built another GLM with each duration (i.e., 0.5, 1, and 2 hours) in each task as a separate condition (i.e., six conditions in total). In both GLMs, we involved the task prompt and the countdown as conditions of no interest. The duration of the task prompt and the countdown were set as the duration of the actual presentation (i.e., 5s and 3s, respectively). The duration of all the other conditions was set as 0. The resulting boxcar and stick functions were convolved with the canonical HRF. Head motion parameters and constant variables indicating each of the six runs were included as nuisance regressors. A high-pass filter with a cutoff of 128s was used to remove the low-frequency noise and slow signal drifts.

Second-level fMRI analysis

We performed the group-level one-sample test on the first-level beta images from the univariate contrast and parametric modulation analyses. The statistical inference was performed using the permutation method with PALM toolbox (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/PALM). Five thousand sign flips were performed (Winkler et al., 2014), and a generalized Pareto distribution was fit to model the tail of the permutation distribution for the P values below 0.1 (Knijnenburg et al., 2009; Winkler et al., 2016). We controlled the family-wise error rate using a conventional cluster-forming threshold (i.e., voxel-wise p < 0.001, cluster-level FWE corrected p < 0.05) (Woo et al., 2014). Given that the hippocampus is our primary region of interest, its elongated and thin shape limits the number of contiguous voxels, restricting the formation of large clusters, and functionally independent clusters within the hippocampus may naturally be small, we performed multiple comparison corrections separately for the cortex and the hippocampus. We used the Automated Anatomical Labelling Atlas 3 to define these masks (Rolls et al., 2020). The cortical mask was defined as all the cortical regions in both hemispheres combined, and the hippocampal mask was defined as the hippocampus in both hemispheres combined (i.e., the No.41 and the No. 42 areas combined).

Acknowledgements

This work was supported by the European Research Council (ERC-StG, NOAM 804422) and the Italian Ministry of University and Research (MUR-FARE, MODGET R18WJMSNZF) attributed to R.B. We thank Mattia Silvestri for the help with data collection and Simone Viganò for the discussion.

Additional information

Funding

European Research Council (10.3030/804422)

Italian Ministry of University and Research (MODGET R18WJMSNZF)