Using fMRI data and experience sampling data to map ongoing thought patterns onto brain activity during movie-watching. Left to Right - One sample of participants was scanned while watching movies (Sample 1), and a different set of participants responded to experience sampling probes (Sample 2) while watching the same movies in the laboratory. Decomposition of mDES data into low-dimension experiential patterns using principal component analysis (PCA) produced a set of dimensions that describe experience during movie-watching (a “thought space” within which the dynamics of the movie-watching experience unfold). Word clouds illustrate how the experience sampling questions map onto each dimension that describes this space. In these word clouds, the font size describes their importance (bigger = more important), and the colour describes their polarity (red = positive, blue = negative). Similarly, we created a brain space to describe the movie-watching experience by comparing each moment in the film to validated dimensions of brain variation. For this purpose, we used the dimensions defined from the resting states of the HCP conducted by Margulies [43] (often referred to as gradients): Gradient 1 (Association to Primary cortex), Gradient 2 (Visual to Motor cortex), Gradient 3 (Frontoparietal to Default Mode Networks) and Gradient 4 (Dorsal Attention Network (DAN)/Visual to Default Mode Networks) of brain variation dimensions illustrated by colour to map activity in state space analysis (purple = low, yellow = high) (not shown: Gradient 5 (Lateral Default Mode to Primary sensory cortex)) [43]. Two 3D scatter plots illustrating two examples from our data of how the movie-watching can be seen as two complimentary trajectories through a “Brain Space” (focusing on Gradients 1, 2 and 3, shown at the top) and a “Thought Space” (focusing on“Episodic Knowledge,” “Verbal Detail,” and “Sensory Engagement,” shown at the bottom). The cooler (blue) points occur earlier in the movie clip and the warmer (red) points occur later.

The relationship between how patterns differ across genres of movies and relate to comprehension. Left to Right – The 3D scatterplot shows the average location of each film on three of the four PCA dimensions, “Episodic Knowledge,” “Verbal Detail,” and “Sensory Engagement.” The bar graphs show the average loading on each dimension, with the error bars showing the 95% Confidence Intervals. The plots on the right illustrate the relationship between the mDES dimensions and memory for information in the film. The top barplot shows the average comprehension score on each film with 95% Confidence Intervals error bars. The scatter plots below show the association between mDES components and comprehension. The scatter plot on the left shows the negative linear relationship between the “Intrusive Distraction” thought and memory. The plot on the right shows a positive association with “Sensory Engagement.” The blue line represents the best-fit line, and the shaded area shows the 95% Confidence Intervals.

The application of multi-dimensional experience sampling (mDES) method and relevant time series produced from decomposed mDES thought patterns. Left to Right – The first panel illustrates the mDES method in the laboratory to demonstrate how participants respond to the sixteen items about their thoughts while watching the film on the laboratory computers. The plots on the right summarize the average thought pattern score at each 15-second sampling window across the three movies. The first time series plot illustrates the trajectory of the “Episodic Knowledge” across Little Miss Sunshine, followed by the time course of “Verbal Detail” also across Little Miss Sunshine, with distinct peaks in scores within the 150-250 second range and particularly low scores between the 400-500 second interval. The third plot demonstrates the relatively low and negative scores on “Sensory Engagement” across Citizenfour. Lastly, the final plot highlights the relatively high scores on “Sensory Engagement” throughout 500 Days of Summer, especially across the 150-400 second interval.

ANOVA across sampling bins of each Movie of each Thought Component score

Group-level neural activation patterns associated with each of the dimensions of thought identified in a voxel space analysis. Left to Right - Regions in red are associated with activity corresponding to reports of “Episodic Knowledge,” green regions are associated with “Intrusive Distraction,” areas in purple are associated with “Verbal Detail,” and the regions in orange represent activity associated with “Sensory Engagement.” The bar plot illustrates the directionality of each parameter estimate with error bars representing 95% Confidence Intervals. Corresponding word clouds for each thought pattern are presented on the right for reference (Top to Bottom: “Episodic Knowledge,” “Intrusive Distraction,” “Verbal Detail,” and “Sensory Engagement”).

Brain regions are associated with multiple experiential features during movie-watching. A region of superior temporal cortex is associated with positive reports of thoughts like “Sensory Engagement” and negative reports of thoughts like “Verbal Detail” (coloured green). A region of dorsal visual cortex was associated with both thoughts reported like “Sensory Engagement” and “Episodic Detail” (coloured red). The word clouds in the middle panel show the results of a Neurosynth analysis of the regions, highlighting the most likely functions associated with these regions. The font size describes their importance (bigger = more important), and the colour describes their polarity (darker = positive). The panel on the right shows the results of seed-based functional connectivity analysis of these regions of overlap from a separate resting-state study. Regions in red indicate those connected to the region of visual cortex, regions in green show those linked to auditory cortex, and regions in yellow are common to both spatial maps.

Comparison of the locations of each moment across the movie clip in the (top row) “Thought Space” and the “Brain Space.” Left to Right – 3D scatterplots of the coordinate locations of each thought pattern (“Episodic Knowledge,” “Verbal Detail,” and “Sensory Engagement) and gradients 1-3 (Gradient 1 (Associated – Primary), Gradient 2 (Visual – Somato-motor), Gradient 3 (Frontoparietal – Default) during Citizenfour, Little Miss Sunshine, and 500 Days of Summer. Observations in blue occur earlier during the film, and observations in red occur later in the film. The gradient maps (1-3) and thought pattern word clouds are presented on the right for reference.

Comparison of “State-Space” and Voxel based analyses of “Sensory Engagement” and “Episodic Knowledge” with Gradients. Left to right – The barplots illustrate the associations for the significant models using Gradients 1-5 as explanatory variables and the thought patterns, “Sensory Engagement” and “Episodic Knowledge” as dependent variables. We performed two spin tests to formally compare these results to those using the voxel space analysis (permutation = 2500). The spin tests revealed the location of the cluster of voxels associated with “Sensory Engagement” are located within the sensory regions of Gradient 1, unlike to have occurred by chance, p = .018. In contrast, the location of the cluster of voxels associated with Episodic Knowledge on Gradient 4 was within the null distribution, p = .251. The locations of the relevant clusters in gradient parcel space are presented in the scatter plots (red points indicate the location of parcels from the relevant comparison).