Current and future goals are represented in opposite patterns in object-selective cortex
- Vrije Universiteit Amsterdam, The Netherlands
- University of Amsterdam, The Netherlands
Figures
Figure 1
Trial design.
(A) Experiment 1. On each trial, participants performed two consecutive visual search tasks. The target objects for both search tasks were presented at the start of the trial. One of the objects could either be a cow, dresser or skate (variable template search; four exemplars per category), and was used for the decoding analyses. The other target was always the same flower (constant template search). The order of presentation (constant or variable template) was manipulated between trials, to create the two main conditions – one in which the variable template was currently relevant, the other in which it was prospectively relevant. To this end, a retro-cue (‘1’ or ‘2’) indicated which of the two previously memorized objects was the target in Search 1. The cue was followed by a delay, then the first search display, followed by a second delay and finally the second search display. Thus, in the Current condition, observers first searched for the variable template (cow, dresser, or skate), and then for the constant template (flower), while this order reversed in the Prospective condition. For each search display, participants indicated whether the target object was present or absent using a button press. At the end of each trial and run participants received feedback about their performance. (B) Experiment 2. Here participants were presented with only one object (cow, dresser or skate) as the possible target template for one of two consecutive visual search tasks. Then a retro-cue appeared, when the cue was ‘1’ the memorized object was a current template, for Search 1; cue ‘2’ indicated that the object was a prospective template, for Search 2; finally, when the cue was ‘0’ the memorized item was not a target in either search and thus it was irrelevant in the trial. The remaining search task in Experiment 2 (either Search 2 in the Current condition, or Search 1 in the Prospective and Irrelevant conditions) was a so-called duplicate search task. In this task, butterflies, motorcycles or trees were presented and participants indicated whether or not any one of the exemplars was shown twice in the display. Thus, here no search template could or needed to be prepared. In the irrelevant condition participants only performed the duplicate search task.
Figure 2 with 2 supplements
Within-relevance and Cross-relevance object category decoding in pFs.
(A) Time course of the Within-relevance decoding where the classifier was trained and tested either within the current, or within the prospective conditions and (B) Average decoding accuracy within the time intervals shown by the shaded areas in (A). Decoding accuracy was higher for currently relevant templates (blue) than for prospectively relevant templates (pink) during the Delay and Search 1 intervals, and vice versa during the Search 2 interval. At the same time, the prospective template could still be reliably decoded during the first search, while the no longer relevant target could be decoded during the second search. (C) Time course of the Cross-relevance category decoding where the classifier was trained on current relevance, tested on prospective relevance, or vice versa and (D) Average decoding accuracy within the time intervals as shaded in (C). This resulted in above-chance decoding during the Delay prior to search (suggesting similar representations for current and prospective templates) but below-chance decoding during Search 1 and Search 2 (suggesting partially opposite representations). Shaded blue and pink areas indicate within-subjects s.e.m. Blue and pink horizontal lines at the bottom of the line graphs indicate time points that significantly differ from chance (p < 0.05, uncorrected). In the bar-plots, colored dots indicate individual participant data, N = 24. *p < 0.05, **p < 0.01, ***p < 0.001, ns: not significant.
-
Figure 2—source data 1
Decoding performance for each participant of Experiment 1: includes source code and data to perform statistical analysis and produce Figure 2.
- https://doi.org/10.7554/eLife.38677.008
-
Figure 2—source data 2
Mean BOLD response for each participant of Experiment 1: includes source code and data to perform statistical analysis and produce Figure 2—figure supplement 1.
- https://doi.org/10.7554/eLife.38677.009
-
Figure 2—source data 3
Cross-temporal generalization matrices for each participant: includes source code and data to perform statistical analysis and produce Figure 2—figure supplement 2.
- https://doi.org/10.7554/eLife.38677.010
Figure 2—figure supplement 1
Time course of the Mean BOLD response in area pFs for current and prospective trials of Experiment 1.
There was a small difference in the BOLD response magnitude during the Delay depending on whether the category was currently or prospectively relevant (t(1,23) = 2.15, p = 0.0427, = 0.44). A stronger difference became apparent for Search 1 (t(1,23) = 14.46, p < 0.001, = 1.77) and Search 2 (t(1,23) = −13.08, p < 0.001, = −2.67), where variable template search displays (containing the object categories of interest) elicited a higher response than constant template search displays (with the repeated flower target), probably because the latter was an easier task. Shaded areas indicate within-subjects s.e.m.
Figure 2—figure supplement 2
Cross-temporal generalization matrices for object category decoding as a function of Relevance.
(A) Within relevence classification on the Current condition, (B) Within relevence classification on the Prospective condition, and (C) Cross-relevance classification (averaged for training on current and testing on prospective with the transpose of the reverse training scheme).Red indicates above-chance decoding performance and blue indicates below-chance decoding performance (see Methods). Note that the pattern on the diagonal reflects the classification per TR as in Figure 2. Of additional interest here are the significant off-diagonal clusters as they indicate a generalized representation across time. We observed that target representations for the objects of interest maintained during the Delay period prior to the first search generalized to the search displays that contained that target (i.e. the first search when current, or the second search when prospective), as shown by reliable off-diagonal red clusters. In contrast, off-diagonal blue clusters emerge when the object trained during the delay is prospective during Search 1, or no longer relevant during Search 2, indicating anti-correlated representations. Outlines indicate cluster-based permutation tests with p < 0.05, positive clusters (solid lines), negative clusters (dotted lines), N = 24.
Figure 3
Representational dissimilarity analysis of object representations in pFs.
(A) Representational dissimilarity matrices for the different variable template categories during Delay, Search 1 and Search 2, as a function of relevance (current and prospective). Blue indicates that representations are more similar while red indicates more dissimilar (B) Multidimensional scaling plots of the same similarity values, for the same Delay, Search 1 and Search 2 intervals. The four exemplars within each category are represented with different shapes (squares, triangles, circles and diamonds). The closer in space the more similar the neural representations. As a trial unfolds, object representations move from predominantly object category space during the delay prior to search (e.g. a cow) into predominantly relevance space (e.g. current target) during search, where current and prospective targets of the same category are represented by partly opposite representational patterns. (C) Comparing dissimilarity between Current and Prospective items when they are drawn from the same category versus when they are drawn from different categories. Representations prior to search within the same category are more similar than different categories, but this reverses during the searches. Colored dots indicate individual participant data, **p < 0.01, ***p < 0.001.
-
Figure 3—source data 1
RDM for each participant of Experiment 1: includes source code and data to perform statistical analysis and produce Figure 3.
- https://doi.org/10.7554/eLife.38677.012
Figure 4 with 1 supplement
Within-relevance and cross-relevance object category decoding in pFs.
(A) Time course of within-relevance decoding and (B) Averaged decoding accuracy within the time intervals shown by the shaded areas in A. During the delay, object category decoding was higher for currently relevant objects (blue) than for irrelevant objects (green) with in between decoding accuracy for prospective templates. During Search 1 the current template showed higher decoding accuracy than the prospective template and the irrelevant item. Importantly, the category of the prospective template could also be decoded during the first search, while the irrelevant category was at chance. During Search 2 the prospective (now current) category was clearly decodable while the formerly current (now no longer relevant) category was at chance (C) Time course of cross-relevance decoding and (D) Averaged decoding accuracy within the time intervals shown by the shaded areas in C. Classification was above chance for all decoding combinations during the Delay prior to search, suggesting similar representations for current, prospective and irrelevant objects. In contrast, we observed below-chance decoding during Search 1 and Search 2 for the Current-Prospective (blue) cross-relevance scheme (suggesting systematically different representations); importantly, this was stronger than for Current-Irrelevant (purple; during Search 1). Current-Irrelevant (purple) and Prospective-Irrelevant (green) cross-classification schemes were at chance. Shaded areas indicate within-subjects s.e.m. Blue and pink, purple and green horizontal lines at the bottom of the line graphs indicate time points that significantly differ from chance (p < 0.05, uncorrected). In the bar-plots, colored dots indicate individual participant data, N = 25. *p < 0.05, **p < 0.01, ***p < 0.001, ns: not significant.
-
Figure 4—source data 1
Decoding performance for each participant of Experiment 2: includes source code and data to perform statistical analysis and produce Figure 4.
- https://doi.org/10.7554/eLife.38677.017
-
Figure 4—source data 2
Mean BOLD response for each participant of Experiment 2: includes source code and data to perform statistical analysis and produce Figure 4—figure supplement 1.
- https://doi.org/10.7554/eLife.38677.018
Figure 4—figure supplement 1
Time course of the Mean BOLD response in area pFs of Experiment 2.
There were no significant differences across relevance conditions during the Delay (F(2,48) = 0.80, p = 0.453, = 0.032, see Figure S1C and S1D). There was a reliable main effect of condition during the Search 1 interval (F(2,48) = 13.57, p < 0.001, = 0.361), with the Prospective condition showing a somewhat weaker response than the Current (t(1,24) = 4.32, p < 0.001, d = 0.86) and Irrelevant (t(1,24) = 4.66, p < 0.001, d = 0.93) conditions, although not as pronounced as in Experiment 1. The Current and Irrelevant condition did not differ from each other (t(1,24) = 1.30, p = 0.205, d = 0.26). As in Experiment 1, the pattern reversed for the Search 2 interval, with the Prospective condition showing a stronger BOLD response than the Current condition (t(1,24) =-2.48, p = 0.020, d = −0.496), although again less pronounced than in Experiment 1. Shaded areas indicate within-subjects s.e.m.
Figure 5
Representational dissimilarity analysis of object representations in pFs.
(A,D,G) Representational dissimilarity matrices and (B,E,H) Multidimensional scaling plots of the same similarity values for the different target object categories during Delay, Search 1 and Search 2, as a function of relevance. The four exemplars within each category are represented with different shapes (squares, triangles, circles and diamonds). (A,B) Comparing Current to Prospective: Object representations moved from predominantly object category space (e.g. a cow) during the Delay period to predominantly relevance space during search, where current and prospective targets of the same category were represented by partly opposite representational patterns. (C) Current-Prospective Mean Dissimilarity. During the delay prior to search, representations within the same category are more similar than of different categories; however, the pattern reverses during Search 1 and Search 2, where representations within the same category are more dissimilar than of different categories. (D, E, F) Comparing Current to Irrelevant and (G, H, I) Comparing Prospective to Irrelevant. During the delay prior to search, targets of the same categories were more similar than of different categories, with no opposite representational pattern during search. Colored dots indicate individual participant data, **p < 0.01, ***p < 0.001, ns: not significant.
-
Figure 5—source data 1
RDM for each participant of Experiment 2: includes source code and data to perform statistical analysis and produce Figure 5.
- https://doi.org/10.7554/eLife.38677.020
Tables
Table 1
Percentage correct and Reaction Time (RT) for Current and Prospective conditions in Search 1 and Search 2 (N = 24) as a function of search order.
https://doi.org/10.7554/eLife.38677.003| Current | Prospective | |||
|---|---|---|---|---|
| Search 1 | Search 2 | Search 1 | Search 2 | |
| Template | Variable | Constant | Constant | Variable |
| P. Correct (%) | 82.2 (7.1) | 98.1 (2.2) | 98.0 (2.3) | 76.0 (9.9) |
| RT (ms) | 1387(20) | 772 (21) | 794 (22) | 1411 (22) |
-
Table 1—source data 1
Behavioral data for each participant of Experiment 1.
- https://doi.org/10.7554/eLife.38677.004
Table 2
Percentage correct and RT in Search 1 and Search 2 (N = 25) as a function of condition.
https://doi.org/10.7554/eLife.38677.013| Current | Prospective | Irrelevant | |||
|---|---|---|---|---|---|
| Search 1 | Search 2 | Search 1 | Search 2 | Search 1 | |
| Template | Duplicate | Duplicate | Template | Duplicate | |
| P. Correct (%) | 86.0 (8.3) | 84.3 (6.9) | 83.9 (5.5) | 83.2 (8.4) | 83.4 (6.9) |
| RT (ms) | 1355 (96) | 1478 (114) | 1460 (90) | 1447(113) | 1469 (91) |
-
Table 2—source data 1
Behavioral data for each participant of Experiment 2.
- https://doi.org/10.7554/eLife.38677.014
Additional files
-
Transparent reporting form
- https://doi.org/10.7554/eLife.38677.021
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Current and future goals are represented in opposite patterns in object-selective cortex
eLife 7:e38677.
https://doi.org/10.7554/eLife.38677
