Mapping patterns of thought onto brain activity during movie-watching

Reviewer #1 (Public Review):

Summary:

The authors used a novel multi-dimensional experience sampling (mDES) approach to identify data-driven patterns of experience samples that they use to interrogate fMRI data collected during naturalistic movie-watching data. They identify a set of multi-sensory features of a set of movies that delineate low-dimensional gradients of BOLD fMRI signal patterns that have previously been linked to fundamental axes of cortical organization.

Strengths:

The novel solution to challenges associated with experience sampling offers potential access to aspects of experience that have been challenging to assess. While inventive, I worry that the reliability of the mDES approach is currently under-investigated, making it challenging to interpret the import of the later analyses, which are themselves strong and compelling.

Weaknesses:

The lack of direct interrogation of individual differences/reliability of the mDES scores warrants some pause.

Reviewer #2 (Public Review):

Summary:

The present study explores how thoughts map onto brain activity, a notoriously challenging question because of the dynamic, subjective, and abstract nature of thoughts. To tackle this question, the authors collected continuous thought ratings from participants watching a movie, and additionally made use of an open-source fMRI dataset recorded during movie watching as well as five established gradients of brain variation as identified in resting state data. Using a voxel-space approach, the results show that episodic knowledge, verbal detail, and sensory engagement of thoughts commonly modulate the activation of the visual and auditory cortex, while intrusive distraction modulates the frontoparietal network. Additionally, sensory engagement is mapped onto a gradient from the primary to the association cortex, while episodic knowledge is mapped onto a gradient from the dorsal attention network to the visual cortex. Building on the association between behavioral performance and neural activation, the authors conclude that sensory coupling to external input and frontoparietal executive control is key to comprehension in naturalistic settings.

The manuscript stands out for its methodological advancements in quantifying thoughts over time and its aim to study the implementation of thoughts in the brain during naturalistic movie watching. However, the conceptualization of thoughts remains vague, its distinction from other concepts like attention is unclear, and interindividual differences are not sufficiently addressed, limiting the study's insights into brain function.

Strengths:

(1) The study raises a question that has been difficult to study in naturalistic settings so far but is key to understanding human cognition, namely how thoughts map onto brain activation.

(2) The thought ratings introduce a novel method for continuously tracking thoughts, promising utility beyond this study.

(3) The authors substantiated the effects of thinking from multiple perspectives, using diverse data types, metrics, and analyses.

(4) The figures are highly informative, accessible, and consistent, aiding comprehension.

Weaknesses:

(1) The dimensions of thought seem to distinguish between sensory and executive processing states. However, it is unclear if this effect primarily pertains to thinking. I could imagine highly intrusive distractions in movie segments to correlate with stagnating plot development, little change in scenery, or incomprehensible events. Put differently, it may primarily be the properties of the movies that evoke different processing modes, but these properties are not accounted for. For example, I'm wondering whether a simple measure of engagement with stimulus materials could explain the effects just as much. How can the effects of thinking be distinguished from the perceptual and semantic properties of the movie, as well as attentional effects? Is the measure used here capturing thought processes beyond what other factors could explain?

(2) I'm skeptical about taking human thought ratings at face value. Intrusive distraction might imply disengagement from stimulus materials, but it could also be an intended effect of the movie to trigger higher-level, abstract thinking. Can a label like intrusive distraction be misleading without considering the actual thought and movie content?

(3) A jittered sampling approach is used to acquire thought ratings every 15 seconds. Are ratings for the same time point averaged across participants? If so, how consistent are ratings among participants? High consistency would suggest thoughts are mainly stimulus-evoked. Low consistency would question the validity of applying ratings from one (group of) participant(s) to brain-related analyses of another participant.

(4) Using three different movies to conclude that different genres evoke different thought patterns (e.g., line 277) seems like an overinterpretation with only one instance per genre.

(5) I see no indication that results were cross-validated, and no effect sizes are reported, leaving the robustness and strength of effects unknown.

Reviewer #3 (Public Review):

This study attempted to investigate the relationship between processing in the human brain during movie watching and corresponding thought processes. This is a highly interesting question, as movie watching presents a semi-constrained task, combining naturally occurring thoughts and common processing of sensory inputs across participants. This task is inherently difficult because in order to know what participants are thinking at any given moment, one has to interrupt the same thought process which is the object of study.

This study attempts to deal with this issue by aggregating staggered experience sampling data across participants in one behavioral study and using the population-level thought patterns to model brain activity in different participants in an open-access fMRI dataset.

The behavioral data consist of 120 participants who watched 3 11-minute movie clips. Participants responded to the mDES questionnaire: 16 visual scales characterizing ongoing thought 5 times, two minutes apart, in each clip. The 16 items are first reduced to 4 factors using PCA, and their levels are compared across the different movies. The factors are "episodic knowledge", "intrusive distraction", "verbal detail", and "sensory engagement". The factors differ between the clips, and distraction is negatively correlated with movie comprehension, and sensory engagement is positively correlated with comprehension.

The components are aggregated across participants (transforming single-subject mDES answers into PCA space and concatenating responses of different participants), and are used as regressors in a GLM analysis. This analysis identifies brain regions corresponding to the components. The resulting brain maps reveal activations that are consistent with the proposed mental processes (e.g. negative loading for intrusion in the frontoparietal network, and positive loadings for visual and auditory cortices for sensory engagement).

Then, the coordinates for brain regions that were significant for more than one component are entered into a paper search in neurosynth. It is not clear what this analysis demonstrates beyond the fact that sensory engagement contains both visual and auditory components.

The next analysis projected group-averaged brain activation onto gradients (based on previous work) and used gradient timecourses to predict the behavioral report timecourses. This revealed that high activations in gradient 1 (sensory→association) predicted high sensory engagement, and that "episodic knowledge" thought patterns were predicted by increased visual cortex activations. Then, permutation tests were performed to see whether these thought pattern-related activations corresponded to well-defined regions on a given cluster.

This paper is framed as presenting a new paradigm but it does little to discuss what this paradigm serves, what its limitations are, and how it should have been tested. I assume that the novelty is in using experience sampling from 1 sample to model the responses of a second sample.

What are the considerations for treating high-order thought patterns that occur during film viewing as stable enough to be used across participants? What would be the limitations of this method? (Do all people reading this paper think comparable thoughts reading through the sections?)

How does this approach differ from collaborative filtering, (for example as presented in Chang et al., 2021)?

In conclusion, this study tackles a highly interesting subject and does it creatively and expertly. It fails to discuss and establish the utility and appropriateness of its proposed method.

Luke J. Chang et al. ,Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience.Sci. Adv.7,eabf7129(2021).DOI:10.1126/sciadv.abf7129