To investigate the transformation of visual representations into goal-directed behaviors, we trained three monkeys to report the 3D orientation of a planar surface (Chang et al., 2020a). Specifically, they performed an eight-alternative forced choice (8AFC) tilt discrimination task with planar surfaces presented at different orientations and distances (Figure 2). The orientation was defined by two angular variables (Stevens, 1983; Rosenberg et al., 2013): tilt and slant. Tilt describes which side of the plane was nearest to the monkey and slant describes the rotation in depth (Figure 2A). Planes were presented for 1 s while fixation was maintained on a target at the center of the screen. The monkey then reported the plane’s tilt (the near side) via a saccade to the corresponding choice target, regardless of the slant or distance (Figure 2B).

Figure 2

Download asset Open asset

Behavioral performance was quantified each session (V3A: Monkey L: N = 39; Monkey F: N = 38; Monkey W: N = 14; CIP: Monkey L: N = 26; Monkey F: N = 27) by calculating the distribution of reported tilt errors (∆Tilt = reported tilt – presented tilt) for each non-zero slant and distance (16 conditions) pooled across tilt (Chang et al., 2020a; Chang et al., 2020b). Each error distribution was then fit with a von Mises probability density function (Equation 1) and behavioral sensitivity was quantified as the concentration parameter ($\kappa$) of the fit. To assess how sensitivity depended on the viewing conditions, we ran a linear mixed-effects model with distance, slant, and area as main effects and animal as a random effect. Consistent with our previous findings, sensitivity decreased with distance from fixation and increased with slant (Figure 2C). Correspondingly, behavioral sensitivity significantly depended on distance (p=4.1 × 10^-30) and slant (p=2.2 × 10^-308). There was no significant effect of area (p=0.46), indicating that behavioral performance was similar during the V3A and CIP sessions. To relate this pattern of behavioral sensitivity to V3A and CIP activity, we next characterized the simultaneously recorded neuronal responses.

The visual system is thought to turn ambiguous 2D retinal signals into behaviorally relevant 3D object representations through a series of transformations. We therefore hypothesized that CIP would contain a higher-level representation of 3D pose than V3A. For a 3D pose selective neuron, the shape of its 3D orientation tuning curve will be tolerant to distance, but its overall response amplitude (gain) should be distance-dependent (Janssen et al., 2000; Nguyenkim and DeAngelis, 2003; Alizadeh et al., 2018; Chang et al., 2020b). In contrast, a neuron selective for lower-level visual features (e.g., binocular disparity) will have 3D orientation tuning curves whose shape and gain are highly distance-dependent. To test for 3D pose tuning, we therefore assessed how 3D orientation tuning depended on distance.

The 3D orientation tuning curves of four representative V3A neurons are shown in Figure 3A–D (qualitatively similar examples from CIP are shown in Figure 3 of Chang et al., 2020b). Some neurons had similar 3D orientation tuning across all (Figure 3A) or most (Figure 3B) distances with distance-dependent gain changes, implying 3D pose tuning. Others had significant orientation tuning at a single distance (ANOVA, p<0.05; Bonferroni–Holm corrected for four distances; Figure 3C), which may reflect intermediate selectivity for gradients of absolute binocular disparity (Nguyenkim and DeAngelis, 2003). The orientation tuning of other neurons changed substantially with distance (Figure 3D), implying lower-level visual feature selectivity. These examples suggest that V3A contains a heterogeneous population of neurons whose functional properties range from processing low-level visual features to 3D object pose.

Figure 3 with 2 supplements see all

Download asset Open asset

In both areas, more neurons had 3D orientation tuning at 57 cm (fixation distance) than at the other distances (Figure 3E). Although the proportion of neurons with significant tuning at each distance was greater in CIP than V3A, the cross-area difference was not significant (chi-squared test, across animals: χ² = 2.4, p=0.50; Monkey L: χ² = 0.62, p=0.89; Monkey F: χ² = 1.6, p=0.67). However, CIP neurons were typically tuned for 3D orientation at more distances than V3A neurons (chi-squared test, across animals: χ² = 21.2, p=2.9 × 10^–4; Monkey L: χ² = 13.0, p=0.01; Monkey F: χ² = 8.7, p=0.07), implying greater convergence of orientation information across distance within CIP than V3A (Figure 3F).

We next examined the 3D orientation preferences and tuning curve shapes by fitting each significant orientation tuning curve (ANOVA, p<0.05; Bonferroni–Holm corrected for four distances) with a Bingham function (Bingham, 1974). The Bingham function is a low-dimensional, parametric model over tilt and slant that describes V3A and CIP 3D orientation tuning curves (Rosenberg et al., 2013; Rosenberg and Angelaki, 2014a; Elmore et al., 2019; Chang et al., 2020b). The preferred orientation taken from these fits is marked with a black dot for the example neurons in Figure 3A–D. In both V3A and CIP, the full span of 3D orientations was represented at each distance (Figure 3—figure supplement 1), indicating that both areas can support neural codes for 3D pose. To compare the shape of the orientation tuning curves, we used the Bingham parameters describing the bandwidth (λ₂), isotropy (λ₁), and axis about which any anisotropy occurred (Φ). First, there was a slight but significant tendency for V3A neurons (median λ₂ = 0.80) to be more narrowly tuned than CIP neurons (median λ₂ = 0.65; linear mixed-effects model with area, absolute distance from fixation, and animal as fixed effects, and neuron as a random effect, p=1.5 × 10^–3; Figure 3—figure supplement 2A and B). This difference may reflect convergent input from multiple V3A neurons onto individual CIP neurons. The tuning bandwidths also increased with distance from fixation (p=6.5 × 10^–6), implying information loss that mirrored the behavioral finding that tilt discrimination performance decreased with distance from fixation (Chang et al., 2020a; Chang et al., 2020b; Figure 2, see Figure 5). No difference was observed across animals (p=0.77). Second, the V3A tuning curves were less isotropic (more elongated; median λ₁ = –1.62) than the CIP tuning curves (median λ₁ = – 0.92), and the difference was significant (p=1.3 × 10^–9; Figure 3—figure supplement 2C and D). The level of anisotropy also significantly increased with distance from fixation (p=9.4 × 10^–6). No difference was observed across animals (p=0.78). Lastly, the distributions of Φ peaked at approximately 90° in both V3A (median Φ = 88°) and CIP (median Φ = 89°), indicating that any anisotropy in the tuning curves generally occurred along the tilt/slant axes (Figure 3—figure supplement 2E and F). These findings indicate greater orientation tuning symmetry in CIP than V3A, which may be important for perceptual sensitivity to changes in object orientation to not depend on the axis of rotation (Chang et al., 2020b).

A previous study found that 3D orientation was better discriminated based on the responses of individual V3A than CIP neurons (Elmore et al., 2019). To follow up on that finding, we computed a surface orientation discrimination index (SODI) that quantifies the difference in responses to preferred and non-preferred orientations relative to the response variability (Equation 2). Neurons with stronger 3D orientation selectivity have SODI values closer to one, whereas those with weaker selectivity have values closer to zero. For each neuron, we calculated the SODI at each distance with significant orientation tuning. In both areas, the mean SODI had an inverted U-shape as a function of distance that peaked at 57 cm (fixation distance; Figure 3G). This indicates that 3D orientation was most discriminable at the fixation distance, which may be a downstream consequence of V1 neurons tending to prefer smaller binocular disparities (Prince et al., 2002). Consistent with the Elmore et al., 2019 study, which measured 3D orientation tuning at the fixation distance only, we found that the SODI values across animals were significantly larger in V3A than CIP at the fixation distance (ANOVA followed by Tukey’s HSD test, p=1.1 × 10^–5). However, the differences were not significant at any other distance (p≥0.47). For Monkey L, the SODI values were significantly larger in V3A than CIP at 37, 57, and 97 cm (p≤8.5 × 10^–3), but not 137 cm (p=0.95). For Monkey F, they were not significantly different at any distance (p≥0.52). Taken together with the Elmore et al., 2019 findings, these results are consistent with a cross-area difference that, as we consider next, may reflect a transformation from lower-level visual feature selectivity to higher-level 3D pose tuning.

To test for cross-area differences in lower-level visual feature selectivity versus higher-level 3D pose tuning, we assessed how the 3D orientation tuning curves depended on distance (Janssen et al., 2000; Nguyenkim and DeAngelis, 2003; Alizadeh et al., 2018). This approach recently revealed 3D pose tuning in CIP (Chang et al., 2020b) but has not been applied to V3A. It thus remains unknown if 3D pose tuning in CIP is simply inherited or reflects a qualitative transformation of feedforward input.

To quantify the distance-dependence of 3D orientation tuning curve shape, we fit each 3D pose tuning curve with a separable model (Equation 3) and computed a tolerance index (Chang et al., 2020b). Tolerance values near zero indicate that the shape of the orientation tuning curve changed substantially with distance (as expected for neurons selective for low-level visual features). Values near one indicate that the shape changed minimally with distance (implying 3D pose tuning). As shown for the example neurons, larger tolerance values were associated with 3D pose tuning (tolerance = 0.96, 0.74; Figure 3A and B, respectively), modest values with more intermediate representations (tolerance = 0.41; Figure 3C), and low values with low-level feature selectivity (tolerance = 0.17; Figure 3D). Across the V3A population, the tolerance values revealed a heterogeneous population in which neurons ranged from having low-level visual feature selectivity to high-level 3D pose tuning (Figure 4A, orange bars).

Figure 4

Download asset Open asset

To test our hypothesis that 3D pose tuning would be more prevalent in CIP than V3A, we performed two complementary analyses. First, we evaluated if there was a cross-area difference in the extent to which the shape of the 3D orientation tuning curves depended on distance by comparing the tolerance distributions. In V3A, the mean tolerance was 0.57 ± 6.7 × 10^–3 SEM across animals (Figure 4A, orange bars; N = 692). Individually, the mean tolerances in V3A were 0.56 ± 8.5 × 10^–3 (Monkey L, N = 311), 0.58 ± 0.01 (Monkey F, N = 263), and 0.55 ± 0.02 (Monkey W, N = 118). In CIP, the mean tolerance was 0.66 ± 7.7 × 10^–3 SEM across animals (Figure 4A, blue bars; N = 437). Individually, the mean tolerances in CIP were 0.65 ± 0.01 (Monkey L, N = 218) and 0.66 ± 0.01 (Monkey F, N = 219). As predicted, the tolerance values in CIP were significantly larger than in V3A (two-sample t-test, across animals: p=7.4 × 10^–19; Monkey L: p=4.7 × 10^–11; Monkey F: p=3.7 × 10^–7), indicating that the shape of 3D orientation tuning curves was more similar across distance in CIP than V3A. Second, we evaluated if there was a cross-area difference in the extent to which the orientation preferences (independent of other tuning parameters such as bandwidth) depended on distance. For each neuron, we calculated the angular deviation between the preferred orientation at each distance and the principal orientation (Chang et al., 2020b; ‘Materials and methods’). Across neurons, we then computed cumulative density functions over the angular deviations (Figure 4B) and found that the deviations were significantly greater in V3A than CIP (Kolmogorov–Smirnov test, across animals: p=1.2 × 10^–9; Monkey L: p=5.1 × 10^–6; Monkey F: p=5.9 × 10^–4). Thus, although both areas represented the full span of 3D orientations at each distance (Figure 3—figure supplement 1), the orientation preferences of individual neurons were more similar across distance in CIP than V3A. These analyses together suggest that a transformation from lower-level visual features to higher-level 3D object representations occurs between V3A and CIP.

We previously found that behavioral tilt sensitivity, which increases as a function of slant and has an inverted U-shape pattern as a function of distance from fixation (Chang et al., 2020a; Figure 2C and black curves in Figure 5), is correlated with neuronal tilt discriminability in CIP (Chang et al., 2020b). To test if a functional correlation between behavior and neuronal activity also exists for V3A, we calculated a tilt discrimination index (TDI; Equation 2) at each slant–distance combination for each neuron, following Chang et al., 2020b. Analogous to the SODI, the TDI quantifies the difference in responses to preferred and non-preferred tilts relative to the response variability. Notably, the mean TDI values in V3A followed the same trends as the behavioral sensitivity for slant (Figure 5A) and distance (Figure 5B). Indeed, the behavioral tilt sensitivities and TDI values were highly correlated across all 16 slant–distance combinations (Monkey L: Spearman r = 0.92, p=2.2 × 10^–308; Monkey F: r = 0.98, p=2.2 × 10^–308; Monkey W: r = 0.74, p=1.5 × 10^–3). Thus, neuronal tilt discriminability in both V3A and CIP was functionally correlated with the 3D tilt sensitivities of the monkeys across a wide range of viewing conditions.

Figure 5

Download asset Open asset

Previous studies found that roughly half of CIP neurons carried choice-related activity during 3D orientation discrimination tasks (Elmore et al., 2019; Chang et al., 2020b). In contrast, one of those studies also reported that choice-related activity was essentially non-existent in V3A, but only tested 23 neurons (Elmore et al., 2019). Given that choice-related activity is preferentially carried by CIP neurons with robust 3D pose tuning (Chang et al., 2020b), the dearth of V3A choice-related activity in the Elmore study could have occurred if the small sample mostly included neurons with low-level feature selectivity. Because that study was not designed to distinguish between low-level visual feature selectivity and 3D pose tuning, we wanted to reassess if V3A carries choice-related activity.

To dissociate choice-related and orientation-selective activity, we analyzed responses to frontoparallel planes (S = 0° and tilt undefined, making them task ambiguous) only. To remove any distance-related response differences, we z-scored the responses at each distance and then pooled across distance. The responses were then grouped according to the monkey’s reported tilt. We first computed eight population-level time courses aligned to the tilt choice that elicited the maximum response for each neuron (Figure 6A). Following an initial transient response, the eight time courses began to separate and showed parametric tuning with amplitudes that symmetrically fell off from the preferred choice (note the similarity of the ±45°, ±90°, and ±135° time courses), thus revealing choice-related activity in V3A. The onset of choice-related activity was defined as the first time point that the eight time courses significantly diverged (191 ms; ANOVA, p<0.05). In contrast, the onset of choice-related activity in CIP was 202 ms (see Figure 5A in Chang et al., 2020b). The finding that choice-related activity appeared first in V3A may reflect that choice signals in CIP contain a large bottom-up contribution from V3A.

Figure 6

Download asset Open asset

Across the populations, 172 (25%) V3A and 201 (46%) CIP neurons carried significant choice-related activity (ANOVA, p<0.05; Figure 6B, inset bar plot). The proportion of neurons with choice-related activity varied across monkeys but was always more prevalent in CIP (Monkey L: N = 72, 33%; Monkey F: N = 129, 59%) than V3A (Monkey L: N = 23, 7%; Monkey F: N = 127, 48%; Monkey W: N = 22, 19%). To test if the strength of choice-related activity differed between the areas, we computed a choice discrimination index (CDI; Equation 2) for each neuron with significant choice tuning (Appendix 1 shows that this index is unaffected by z-scoring, which was used in calculating the choice tuning curves). The CDI values were highly similar in V3A (mean CDI = 0.38 ± 5.6 × 10^–3 SEM, N = 172) and CIP (mean CDI = 0.38 ± 5.7 × 10^–3 SEM, N = 201) and not significantly different (Wilcoxon rank-sum test, p=0.20). Thus, although there was a cross-area difference in the prevalence of choice-related activity, it was nevertheless present in V3A and similar in strength to CIP.

One potential reason that choice-related activity was more prevalent in CIP than V3A is a difference in the structure of correlated variability between the areas (Nienborg and Cumming, 2006). To test this, we computed noise and signal correlations using the 3D pose data for pairs of neurons simultaneously recorded on the same tetrode (‘Materials and methods’). The noise correlations were almost identical in V3A (across animals: mean r = 0.17 ± 0.01 SEM, N = 404 pairs; Monkey L: 0.15 ± 0.01, N = 206; Monkey F: 0.15 ± 0.03, N = 121; Monkey W: 0.25 ± 0.03, N = 77) and CIP (across animals: 0.17 ± 0.01, N = 244 pairs; Monkey L: 0.14 ± 0.02, N = 107; Monkey F: 0.19 ± 0.02, N = 137), and not significantly different (Wilcoxon rank-sum test, across animals: p=0.62; Monkey L: p=0.58; Monkey F: p=0.49; Figure 6B, right marginal histogram). However, noise correlations alone are not sufficient to account for differences in choice-related activity. Instead, the relationship between noise and signal correlations matters (Liu et al., 2013; Cumming and Nienborg, 2016). We therefore quantified the signal correlations for the same pairs of neurons and found that they were also similar in V3A (across animals: mean r = 0.40 ± 0.02 SEM; Monkey L: 0.32 ± 0.03; Monkey F: 0.41 ± 0.04; Monkey W: 0.61 ± 0.04) and CIP (across animals: 0.36 ± 0.03; Monkey L: 0.26 ± 0.04; Monkey F: 0.43 ± 0.03), and not significantly different (across animals: p=0.15; Monkey L: p=0.32; Monkey F: p=0.86; Figure 6B, top marginal histogram). We then compared the relationship between noise and signal correlations (Figure 6B). As expected, higher noise correlations were associated with higher signal correlations (Shadlen et al., 1996; Cohen and Maunsell, 2009; Gu et al., 2011). Importantly, the linear relationship between noise and signal correlation was not significantly different between the areas (ANCOVA, across animals: p=0.37; Monkey L: p=0.06; Monkey F: p=0.30). Furthermore, between pairs of neurons within each area in which (i) both had choice-related activity, (ii) neither had choice-related activity, or (iii) one had but the other did not have choice-related activity, we found no significant differences in the magnitudes of noise (Kruskal–Wallis test, both p≥0.15) and signal (both p≥0.19) correlations or their linear relationship (ANCOVA, both p≥0.23). These results suggest that differences in the structure of correlated variability either within or across areas did not account for the presence of choice-related activity. As such, the cross-area difference in prevalence of choice-related activity was consistent with a stronger functional correlation between neuronal activity and 3D perceptual decisions in CIP than V3A.

Choice tuning curves are shown for the four example V3A neurons from Figure 3 in Figure 6C–F. The two neurons with higher tolerances (Figure 3A and B) both carried choice-related activity (Figure 6C and D, purple curves). Notably, the tilt tuning curves (marginalized over slant and distance; Figure 6C and D, black curves) were well aligned with the choice tuning curves. In contrast, the neurons with intermediate and low tolerances (Figure 3C and D) did not carry choice-related activity (Figure 6E and F, purple points). To quantify the relationship between tilt and choice preferences for neurons with significant orientation and choice tuning, we took the circular difference between each neuron’s preferred tilt (from the principal orientation) and preferred tilt choice (from the von Mises fit). In both areas, the median circular difference between the preferences (V3A: 1.57°, N = 168; CIP: –0.75°, N = 194) was not significantly different from 0° (circular median test, both p≥0.49), indicating that the tilt and choice preferences tended to align (Figure 6G). To assess if the relationship between tilt and choice preferences differed between the areas, we compared the widths of the two distributions and found that they had similar circular variance (CV) values (V3A: CV = 0.34; CIP: CV = 0.33), which were not significantly different (two-sample concentration difference test, p=0.88) (Fisher, 1995). Thus, the strength of the association between tilt and choice preferences was similar in V3A and CIP.

We previously found that CIP neurons with more robust 3D pose tuning (higher tolerance values) preferentially carried choice-related activity (Chang et al., 2020b). To assess this relationship in V3A, we compared the tolerance values of V3A neurons with and without choice-related activity (Figure 7A). Indeed, V3A neurons with choice-related activity had a mean tolerance of 0.65 ± 1.3 × 10^–2 SEM (N = 172), whereas those without choice-related activity had a mean tolerance of 0.54 ± 7.2 × 10^–3 SEM (N = 520), and the difference was significant (ANOVA followed by Tukey’s HSD test, p=3.8 × 10^–9). Thus, choice-related activity was preferentially carried by V3A neurons with more robust 3D pose tuning, as in CIP.

Figure 7

Download asset Open asset

Because more robust 3D pose tuning was associated with choice-related activity, we were concerned that the difference in 3D selectivity between V3A and CIP (Figure 4) simply reflected a cross-area difference in the prevalence of choice signals. However, this was not the case. First, tolerance values were greater in CIP than V3A both for neurons with (ANOVA followed by Tukey’s HSD test, p=1.7 × 10^–7) and without (p=1.6 × 10^–4) choice-related activity (Figure 7B). Second, comparisons of the cumulative density functions over the angular deviations between orientation preferences at each distance and the principal orientation (Figure 7C) revealed significantly smaller deviations for neurons with than without choice-related activity in both areas (Kolmogorov–Smirnov test, p≤1.1 × 10^–5). The deviations were also significantly smaller in CIP than V3A both for neurons with (p=1.3 × 10^–4) and without (p=8.8 × 10^–4) choice-related activity. Thus, 3D pose tuning was more robust in CIP than V3A, regardless of whether the neurons carried choice-related activity.

Previous studies reported that V3A contains extraretinal signals associated with creating stable, allocentric representations of the world (Galletti and Battaglini, 1989; Galletti et al., 1990; Sauvan and Peterhans, 1999; Nakamura and Colby, 2002), as well as postsaccadic activity (Nakamura and Colby, 2000). Recently, we discovered saccade-related activity in CIP that predicted the direction and timing of eye movements (Chang et al., 2020b). We therefore hypothesized that V3A may possess similar saccade-related activity. To test this possibility, we trained the monkeys to perform a visually guided (pop-up) saccade task (Munoz and Wurtz, 1995; Hanes and Schall, 1996; Figure 8A).

Figure 8 with 1 supplement see all

Download asset Open asset

To first determine if V3A carries saccade-related activity, we computed eight population-level time courses relative to the saccade direction that elicited the maximum response for each neuron (Figure 8). Noting that the time courses were parametrically tuned, we calculated the start of the activity by finding the first time point at which they significantly diverged (ANOVA, p<0.05). Intriguingly, saccade-related activity started in V3A (a classically defined ‘intermediate’ visual area) 108 ms prior to saccade initiation, which was earlier than in CIP (102 ms; see Figure 8B in Chang et al., 2020b). In both areas, the time course of saccade-related activity was distinct from the visually evoked response measured during receptive field mapping, consistent with presaccadic activity (Figure 8—figure supplement 1; see ‘Discussion’).

The finding of saccade-related activity in V3A prompted us to test if that activity predicted the timing of the saccades. Each trial in which a saccade was made in the preferred saccade direction was labeled with the saccade latency (the time from target appearance to saccade initiation). The distribution of latencies was then divided into quartiles (Figure 8C) and the time course of saccade-related activity computed for each quartile (Figure 8D). On trials that the saccade latency was shorter, saccade-related activity started closer to the saccade initiation. Indeed, the growth rate (linear slope) from the start of saccade-related activity (ANOVA, p<0.05) to when the four curves approximately coalesced (59 ms before saccade initiation; ‘Materials and methods’) was highly correlated and inversely related to the mean saccade latency (r = –0.98, p=1.9 × 10^–2; Figure 8E). Intriguingly, these results indicate that V3A activity predicts the direction and timing of upcoming saccadic eye movements.

We further noted that the V3A time courses were visually more similar to step functions than the CIP time courses, which more closely resembled ramping activity. This informal observation was reflected in steeper growth rates for each quartile in V3A (Q1 = 1.8; Q2 = 1.1; Q3 = 0.82; Q4 = 0.44) than CIP (Q1 = 0.73; Q2 = 0.61; Q3 = 0.56; Q4 = 0.39) and led us to speculate that CIP might temporally integrate saccade-related signals from V3A. For each latency quartile, we therefore integrated the V3A time courses from the onset of V3A activity to when the CIP time courses approximately coalesced (15 ms before saccade initiation; ‘Materials and methods’; Chang et al., 2020b) and compared them to the observed CIP time courses. Notably, the integrated V3A output (Figure 8F, dashed curves) aligned well with the CIP activity (solid curves) for each quartile, as might be expected if CIP accumulates evidence from V3A in favor of particular oculomotor responses. Consistent with CIP being closer to the site of saccade initiation than V3A, the CIP time courses coalesced 15 ms before saccade initiation compared to 59 ms in V3A. These results imply that V3A is not simply a ‘visual area’ and may suggest that there are parallel visual and saccade-related V3A-to-CIP hierarchies.

We next examined the saccade direction tuning of individual V3A neurons. Saccade direction tuning curves for the four example V3A neurons are shown in Figure 9A–D (green curves) along with von Mises fits for those with significant tuning (ANOVA, p<0.05). Across animals, 415 (60%) of the V3A neurons had significant saccade direction tuning (Monkey L: 172, 55%; Monkey F: 154, 58%; Monkey W: 89, 75%), similar to that observed in CIP (across animals: 274, 63%; Monkey L: 153, 70%; Monkey F: 121, 55%) (Chang et al., 2020b). Like CIP neurons, the V3A neurons showed parametric tuning for saccade direction with responses that fell off symmetrically from the preferred direction. Correspondingly, the tuning curves were well described by von Mises functions (mean r = 0.92 ± 0.47 × 10^–3 SEM, N = 415).

Figure 9 with 1 supplement see all

Download asset Open asset

To test for cross-area differences in saccade direction tuning, we first calculated a saccade discrimination index (SDI; Equation 2) for neurons with significant tuning. Surprisingly, saccade direction was more discriminable based on V3A (across animals: mean SDI = 0.49 ± 6.2 × 10^–3 SEM, N = 415; Monkey L: 0.48 ± 0.02, N = 172; Monkey F: 0.51 ± 9.3 × 10^–3, N = 154; Monkey W: 0.48 ± 0.01, N = 89) than CIP (across animals: 0.41 ± 6.2 × 10^–3, N = 274; Monkey L: 0.40 ± 8.7 × 10^–3, N = 153; Monkey F: 0.42 ± 8.7 × 10^–3, N = 121) responses, and the difference was statistically significant (Wilcoxon rank-sum test, across animals: p=4.8 × 10^–17; Monkey L: p=5.1 × 10^–8; Monkey F: p=2.6 × 10^–11). We further compared the tuning bandwidths ($\kappa$ from the von Mises fits) and found that the V3A tuning curves were narrower (across animals: mean $\kappa$ = 6.3 ± 0.26 SEM; Monkey L: 6.1 ± 0.45; Monkey F: 6.6 ± 0.40; Monkey W: 6.2 ± 0.53) than the CIP tuning curves (across animals: 4.7 ± 0.29; Monkey L: 4.4 ± 0.38; Monkey F: 5.0 ± 0.46), and the difference was statistically significant (Wilcoxon rank-sum test, across animals: p=1.9 × 10^–7; Monkey L: p=0.01; Monkey F: p=8.4 × 10^–5). These results are consistent with convergent input from multiple V3A neurons onto individual CIP neurons and parallel the cross-area difference in 3D orientation tuning.

We additionally assessed if choice- and saccade-related activity were functionally dissociable in V3A as they are in CIP (Chang et al., 2020b). Across the V3A population, 124 neurons (18%) had both choice- and saccade-related activity. For this subpopulation, choice and saccade preferences generally aligned (Figure 9E). The median circular difference (–1.7°) between the preferences was not significantly different from 0° (circular median test, p=0.65). Although the preferences aligned, other characteristics were distinct. First, the saccade tuning curves were narrower (mean $\kappa$ = 6.3 ± 0.26 SEM) than the choice tuning curves (mean $\kappa$ = 4.7 ± 0.46 SEM), and the difference was significant (Wilcoxon rank-sum test, p=6.4 × 10^–11). Second, some neurons carried choice- (48, 7%) or saccade-related (291, 42%) activity only, indicating that the properties were not mutually inclusive. Third, choice-related activity was associated with more robust 3D tuning, whereas saccade-related activity was associated with less robust 3D tuning (Figure 9—figure supplement 1), indicating that choice and saccade signals had opposite functional relationships with 3D selectivity in V3A. These results together suggest that the choice- and saccade-related activities were functionally distinct.

Neurons in CIP form associations between their surface orientation and choice/saccade direction preferences such that the alignment of the tuning curves reflects whether the monkey was trained to report the near or far side of the plane (Elmore et al., 2019; Chang et al., 2020b). However, the existence of a similar sensorimotor association was not immediately evident based on the saccade direction and tilt tuning curves (marginalized over slant and distance) of the example V3A neurons (Figure 9A–D). Assuming that decision-related processing occupies an intermediate position between sensory and motor activity, it is possible that choice signals have a moderating effect on the sensorimotor association that could obscure the relationship if not taken into consideration. Indeed, we previously found that the strength of the sensorimotor association in CIP depended on the presence of choice signals (Chang et al., 2020b).

To test for sensorimotor associations in V3A and evaluate the potential moderating effect of choice signals, we therefore calculated the angular difference between the principal tilt (from the principal orientation) and saccade direction preference for neurons with and without choice-related activity (Figure 9F, left, orange and gray bars, respectively). For neurons without choice-related activity, there was no discernible association between the tilt and saccade direction preferences since the distribution was not significantly different from uniform (Rayleigh test, p=0.08, N = 270). However, for neurons with choice-related activity, the distribution was significantly different from uniform (p=1.2 × 10^–18, N = 122) and the median circular difference (–1.6°) was not significantly different from 0° (circular median test, p=0.42), revealing a sensorimotor association. Correspondingly, the distribution of preference differences was significantly narrower for neurons with (CV = 0.44) than without (CV = 0.9) choice-related activity (two-sample concentration difference test, p=1.4 × 10^–5; Figure 9G). Thus, a sensorimotor association was only evident for V3A neurons with choice-related activity.

When we repeated this analysis for CIP, we found a striking difference from V3A (Figure 9F, right). Specifically, sensorimotor associations were evident regardless of whether the neurons carried choice signals. The distributions of preference differences were significantly different from uniform both for neurons with (blue bars; p=3.7 × 10^–29, N = 134) and without (gray bars; p=9.7 × 10^–5, N = 122) choice-related activity. Likewise, the median differences were not significantly different from 0° both for neurons with (–4.2°, p=0.14) and without (–3.2°, p=0.65) choice-related activity. This reveals that choice signals were not necessary for sensorimotor associations in CIP, but the distribution was significantly narrower for neurons with (CV = 0.34) than without (CV = 0.73) choice-related activity (two-sample concentration difference test, p=2.6 × 10^–5; Figure 9G), indicating that sensorimotor associations were strongest for neurons with choice-related activity.

We further assessed the cross-area difference in the overall strength of sensorimotor associations by comparing the widths of the V3A and CIP preference difference distributions (including neurons with and without choice-related activity). The distribution was significantly broader in V3A (CV = 0.77) than CIP (CV = 0.53; p=1.2 × 10^–3), indicating that sensorimotor associations were strongest in CIP (Figure 9G). These results thus imply a hierarchical transformation in the strength of sensorimotor associations along the ‘where’ pathway and suggest a novel role for choice-related activity in sensorimotor processing as explored next.

The above analyses showed that neurons with choice-related activity exhibited more robust 3D pose tuning (Figure 7) and formed stronger sensorimotor associations (Figure 9G). If choice signals occupy an intermediate position between sensory and motor processing, the strength of sensorimotor associations might depend on the robustness of 3D tuning for neurons with choice-related activity but not those without choice-related activity. In other words, choice-related activity may statistically moderate (Judd et al., 2014) the relationship between the robustness of 3D pose tuning and the strength of sensorimotor associations. To test this, we ran a linear regression model where the absolute angular difference between the principal tilt and saccade direction preference depended on tolerance, choice-related activity, and their interaction. Both areas showed a significant interaction (both p≤1.6 × 10^–3) such that tolerance had a negligible impact on the strength of the sensorimotor association for neurons without choice-related activity (V3A: slope = 0.16; CIP: slope = –0.15) but a strong impact for neurons with choice-related activity (V3A: slope = –2.2; CIP: slope = –2.5). Thus, this analysis revealed an intricate relationship between sensory, choice, and saccade-related activity in which choice signals moderated the relationship between the robustness of 3D tuning and sensorimotor associations.

Multiple lines of evidence converged to support a V3A-to-CIP hierarchy. Focusing first on the visual properties, we found that the median visual response latency in V3A was 6 ms shorter than in CIP and that the receptive fields were smaller in V3A than CIP. At a functional level, 3D orientation and position information (which are confounded in retinal images) were more separable in CIP than V3A, implying a hierarchical resolution of sensory ambiguities that limit the ability to make 3D perceptual inferences.

Intriguingly, our findings may also reveal a V3A-to-CIP hierarchy related to oculomotor processing. While saccade-related activity that predicted the direction and timing of eye movements was prevalent in both areas, it began 6 ms earlier in V3A than CIP (in agreement with the difference in visual response latencies). The finding that the time course of saccade-related activity in CIP closely resembled the temporally integrated V3A output may further suggest that some presaccadic signals originate in a region of visual cortex whose feedforward input includes V1 and V2 (Zeki, 1980; Felleman and Van Essen, 1991), thus challenging classical notions of sensorimotor dichotomies. This finding is also consistent with CIP accumulating evidence provided by V3A in favor of particular oculomotor responses, which raises the possibility that some of the integration-like properties of LIP (Shadlen and Newsome, 1996; Roitman and Shadlen, 2002) may reflect bottom-up input (Lewis and Van Essen, 2000; Premereur et al., 2015; Van Dromme et al., 2016). In that case, inactivating CIP, which is architectonically distinct from LIP (Katsuyama et al., 2010; Niu et al., 2020), might impair saccadic responses during decision-making tasks. Consistent with this, reversible inactivation of CIP during a depth structure categorization task was found to delay saccadic responses (Van Dromme et al., 2016). While that delay may have reflected degraded visual discrimination, the current findings also point to the possibility of impaired saccade preparation.

While these experiments revealed saccade-related activity in V3A and CIP, the protocol was not designed to dissociate the potential contributions of visual and presaccadic signals. Nevertheless, several lines of evidence suggest that the observed saccade-related activity cannot be explained by visual responses alone. First, the time course of saccade-related activity was distinct from that of the visual flash response. Examination of the time courses provided some evidence that a presaccadic signal may begin in V3A 7 ms before CIP, consistent with the possibility of a V3A-to-CIP oculomotor hierarchy. Second, there was a pronounced relationship between the growth rate of the saccade-related activity and saccade latency. Third, there was an apparent integrative relationship between the V3A and CIP saccade-related activity, which does not exist for the visual responses. Fourth, the sensorimotor association between tilt and saccade direction preferences as well as the moderation by choice-related activity would not be expected for visual flash responses. The current findings therefore suggest that V3A and CIP may contain presaccadic signals, but a thorough evaluation of this possibility will require future studies.

Many V3A neurons showed functional associations between their orientation, choice, and saccade direction preferences, as in CIP (Elmore et al., 2019; Chang et al., 2020b). Finding these associations in V3A implies that sensorimotor processing traditionally linked to parietal cortex already occurs in an ‘intermediate visual area,’ suggesting that V3A may be more appropriately classified as an early association region. Intriguingly, saccade direction discrimination was greater in V3A (where saccade-related activity was associated with poorer 3D pose tuning) while sensorimotor associations were stronger in CIP (where no relationship between saccade-related activity and the robustness of 3D tuning was evident). These cross-area differences may reflect a transition from lower-level representations of saccade-related signals to higher-level sensorimotor associations and parallel the observed changes in visual feature selectivity.

Previous studies arrived at categorically different conclusions regarding visual processing in V3A (Gaska et al., 1987, Gaska et al., 1988; Galletti and Battaglini, 1989; Galletti et al., 1990; Sauvan and Peterhans, 1999; Nakamura and Colby, 2000; Nakamura and Colby, 2002; Tsao et al., 2003; Anzai et al., 2011; Elmore et al., 2019). Our large V3A sample revealed a heterogeneous population in which neurons ranged from representing low-level visual features to high-level object properties. Earlier discrepancies may reflect a combination of this heterogeneity, small sample sizes, and the possibility of functionally distinct modules within V3A. Another factor that may have contributed to the discrepancies is variability in how V3A was defined (Nakhla et al., 2021), highlighting the continued importance of using functional and anatomical localization methods in future investigations of this relatively understudied area.

For some V3A neurons, we found that orientation tuning curve shape (but not gain) was highly tolerant to distance, implying 3D pose tuning. Because the perspective cues in our stimuli were independent of distance, the gain changes must have been driven by stereoscopic cues. This implies that these neurons were selective for gradients of relative disparity. Other V3A neurons showed orientation tuning at a single distance. These neurons may have been selective for absolute disparity gradients, similar to some middle temporal (MT) area neurons (Nguyenkim and DeAngelis, 2003). Such neurons may reflect an intermediate stage of visual processing whose outputs are combined to create 3D pose representations. Selectivity for absolute disparity gradients may further account for suppressive effects previously reported within the classical receptive fields of V3A neurons (Gaska et al., 1987) since the stimuli were presented at screen distance only and therefore would stimulate portions of the receptive field with non-preferred disparities. Lastly, the orientation tuning curve shape of some V3A neurons was highly distance-dependent, which may reflect tuning for a single absolute disparity. Future studies that perform detailed receptive field submapping of disparity selectivity will be important to explicate the heterogeneity of V3A as well as the transformations by which the visual system achieves 3D pose tuning. One possibility is that absolute disparity representations (in V1) are used to construct absolute disparity gradient detectors (in V3A, MT) and then pose selectivity (in V3A, PIP, CIP).

Contemporary interpretations of choice signals include a mix of feedforward contributions to decision processes, the structure of correlated variability, attention, cognitive and behavioral factors, as well as feedback (Celebrini and Newsome, 1994; Britten et al., 1996; Dodd et al., 2001; Haefner et al., 2013; Gu et al., 2014; Ruff and Cohen, 2014; Smolyanskaya et al., 2015; Cumming and Nienborg, 2016). Despite these complexities, the current findings are consistent with a feedforward cascade and build-up of choice signals, which may reflect greater contributions of CIP than V3A to 3D perceptual decisions. First, choice-related activity appeared 11 ms earlier in V3A than CIP. Second, it was about twice as prevalent in CIP as V3A. Notably, the structure of correlated variability could not account for the prevalence of choice-related activity either within or across areas. Third, choice-related activity was preferentially carried by neurons with 3D orientation tuning that was more tolerant to distance. This is consistent with reports that neurons that carry choice signals tend to have resolved sensory ambiguities about the information being discriminated, allowing them to more directly contribute to decisions (Liu et al., 2013; Chang et al., 2020b). Although choice signals may be confounded with feature-based attentional modulation (Cohen and Newsome, 2008), this is unlikely to explain our findings. Attention is largely associated with changes in response magnitude (Reynolds and Heeger, 2009). However, we found more intricate relationships in which choice-related activity (i) was associated with changes in tuning curve shape that made 3D orientation tuning more tolerant to distance (Chang et al., 2020b) and (ii) statistically moderated the strength of sensorimotor associations such that visual and saccade direction preferences were best aligned for neurons that carried choice signals. Indeed, the strength of sensorimotor associations strongly depended on the robustness of 3D pose tuning for neurons with (but not without) choice-related activity. These findings thus reveal a multifaceted landscape of functional associations between sensory, choice-, and motor-related activity. They further imply a novel role for choice signals in sensorimotor processing that might reflect the temporal cascade of sensory processing, response selection, and motor action.

All procedures followed the National Institutes of Health’s Guide for the Care and Use of Laboratory Animals and were approved by the Institutional Animal Care and Use Committee at the University of Wisconsin–Madison (Protocol G005229). Three male rhesus monkeys (Macaca mulatta; Monkey L: 6 years of age; Monkey F: 5 years; Monkey W: 5 years) were surgically implanted with a Delrin ring for head restraint and a removable recording grid for guiding electrodes. After recovery, they were trained to sit in a primate chair with head restraint and to fixate visual targets within 2° version and 1° vergence windows for liquid rewards.

Experimental control was performed using the REC-GUI software (RRID:SCR_019008) (Kim et al., 2019). Stimuli were rendered using Psychtoolbox 3 (MATLAB R2016b; NVIDIA GeForce GTX 970) and rear-projected onto a polarization preserving screen (Stewart Film Screen, Inc) using a DLP LED projector (PROPixx; VPixx Technologies, Inc) with 1280 × 720 pixel resolution (70° × 43° of visual angle) at 240 Hz. A circular polarizer was used to sequence the presentation of stereoscopic ‘half-images’ to each eye (120 Hz/eye). Polarized glasses were worn. A phototransistor circuit was used to confirm the synchronization of the left and right eye images as well as align neuronal responses to the stimulus onset. Eye tracking was performed optically at 1 kHz (EyeLink 1000 plus, SR Research). The monkeys sat 57 cm from the screen.

The stimuli were previously described in detail (Chang et al., 2020a; Chang et al., 2020b). Briefly, planar surfaces subtending 20° of visual angle were presented at the center of the screen. They were defined by 250 nonoverlapping dots that were uniformly distributed across the plane and rendered with stereoscopic and perspective cues. Surface orientation was described using tilt and slant (Stevens, 1983; Rosenberg et al., 2013). All combinations of eight tilts (0–315°, 45° steps) and four slants (15–60°, 15° steps) plus the frontoparallel plane (tilt undefined, slant = 0°) were presented (N = 33). All orientations were presented at four distances (37, 57, 97, and 137 cm; N = 132 unique poses). The dots were scaled with distance such that their screen size only depended on slant. At a slant of 0°, each dot subtended 0.35°.

The fixation point subtended 0.3° and was always at 57 cm (screen distance). Keeping its distance constant while varying the plane’s distance was a key design feature (Nguyenkim and DeAngelis, 2003; Hegdé and Van Essen, 2005; Ban and Welchman, 2015; Alizadeh et al., 2018; Henderson et al., 2019) that conferred two advantages over yoking the distance of the stimulus and fixation point (Banks et al., 2001; Hillis et al., 2004). First, it ensured that the monkeys could not rely on local absolute disparity cues to judge 3D orientation (Elmore et al., 2019; Chang et al., 2020a; Chang et al., 2020b). Second, it allowed us to dissociate the effects of stimulus distance and vergence signals (which would have otherwise been confounded) on the neuronal responses, which is important because V3A and CIP carry extraretinal signals.

Tilt discrimination performance was quantified by calculating the distribution of reported tilt errors (ΔTilt = reported tilt – presented tilt) and fitting a von Mises probability density function:

The mean (µ) and concentration ($\kappa$) parameters describe the accuracy and sensitivity, respectively (Seilheimer et al., 2014; Dakin and Rosenberg, 2018). Values of µ closer to 0° indicate greater accuracy and larger $\kappa$ indicate greater sensitivity. An upper bound of $\kappa$ = 18 was set in the estimation routine (Chang et al., 2020a; Chang et al., 2020b). A modified Bessel function of order 0, $I_0\left(\kappa \right)$, normalizes the function to have unit area.

To target the areas, magnetic resonance imaging (MRI) scans were collected on a 3-Tesla GE MR750 scanner before and after implanting the head restraint ring. To estimate the penetration trajectories, the CARET software was used to register the structural scans to the F99 atlas (Van Essen et al., 2001) and align the recording grid (Rosenberg et al., 2013; Rosenberg and Angelaki, 2014a; Rosenberg and Angelaki, 2014b; Chang et al., 2020b). During penetrations, observed gray/white matter and sulcal transitions were referenced to the MRI. Area CIP is located in the caudal portion of the lateral bank of the intraparietal sulcus and ventral to LIP. Area V3A is located adjacent to and ventral-laterally to CIP. There is a swath of white matter dorsal to V3A and lateral to CIP.

The order of the recording sessions went as follows. First, 41 V3A sessions were performed (Monkey L: N = 21; Monkey F: N = 20). Then, all 53 CIP sessions were performed (Monkey L: N = 26; Monkey F: N = 27). Finally, 50 more V3A sessions were performed (Monkey L: N = 18; Monkey F: N = 18; Monkey W: N = 14). The recordings were performed using linear array probes with either four or eight tetrodes (NeuroNexus, Inc). The tetrodes were separated by 300 μm and electrodes within a tetrode were separated by 25 μm. Neuronal signals (sampled at 30 kHz) along with eye position and phototransistor signals (each sampled at 1 kHz) were stored using a Scout Processor (Ripple, Inc). Tetrode-based spike sorting was performed offline using the KlustaKwik (K. Harris) semi-automatic clustering algorithm in MClust (MClust-4.0, A.D. Redish et al.) followed by manual refinement using Offline Sorter (Plexon, Inc). Only well-isolated single neurons verified by at least two authors were included.

Initial receptive field (RF) estimates were made by hand-mapping with patches of random dots, sinusoidal gratings, and/or orientated bars. During hand-mapping, fixation was maintained on a target (0.3°) at the center of the screen and a liquid reward was provided every 2–3 s of continuous fixation. Breaks in fixation resulted in the disappearance of the fixation point and stimulus for 1 s. An automated stimulus in which bright and dark squares were flashed one square at a time was then presented on a gray background. For V3A, 2° × 2° squares tiled a region centered on and larger than the hand estimate. For CIP, the squares were 4° × 4° and tiled the entire screen. A monkey was required to first fixate a target (0.3°) at the center of the screen for 300 ms. Then, while maintaining fixation, the squares were flashed (150 ms duration) in an alternating sequence at pseudorandom locations. A liquid reward was provided after every 20 flashes. If fixation was broken, the monkey was required to again fixate the target for 300 ms before the sequence resumed. A single repetition was completed when both bright and dark squares had been flashed at every location. At least five repetitions were collected each session.

The RF boundary was estimated offline by calculating the firing rate at each tiled location, averaging over the bright and dark square responses. The responses at each location were then compared to baseline, calculated using the last 150 ms of the fixation periods preceding the stimulus sequences, to identify significant responses (ANOVA, p<0.05). The RF map was then manually smoothed and an envelope contour drawn. This procedure produced RF estimates for 355 (51%) V3A and 112 (26%) CIP neurons. The cross-area difference in the proportion of neurons for which a RF could be estimated may reflect that the mapping stimulus was likely more appropriate for lower- than higher-level visual areas.

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

We thank Zikang Zhu for helpful discussion, as well as Meghan Lowe and Satchal Postlewaite for help with spike sorting. Further support was provided by the National Institutes of Health Grant P51OD011106 to the Wisconsin National Primate Research Center and the National Institute of Child Health and Human Development Grant P50HD105353 to the Waisman Center.

This study was performed in strict accordance with the recommendations of the National Institutes of Health's Guide for the Care and Use of Laboratory Animals. All experimental procedures and surgeries were approved by the Institutional Animal Care and Use Committee (IACUC) at the University of Wisconsin-Madison (Protocol #: G005229).

© 2022, Doudlah, Chang et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.