1. Neuroscience
Download icon

Stimulus vignetting and orientation selectivity in human visual cortex

  1. Zvi N Roth  Is a corresponding author
  2. David J Heeger
  3. Elisha P Merriam
  1. National Institute of Mental Health, National Institutes of Health, United States
  2. New York University, United States
Research Article
  • Cited 1
  • Views 1,194
  • Annotations
Cite this article as: eLife 2018;7:e37241 doi: 10.7554/eLife.37241

Abstract

Neural selectivity to orientation is one of the simplest and most thoroughly-studied cortical sensory features. Here, we show that a large body of research that purported to measure orientation tuning may have in fact been inadvertently measuring sensitivity to second-order changes in luminance, a phenomenon we term ‘vignetting'. Using a computational model of neural responses in primary visual cortex (V1), we demonstrate the impact of vignetting on simulated V1 responses. We then used the model to generate a set of predictions, which we confirmed with functional MRI experiments in human observers. Our results demonstrate that stimulus vignetting can wholly determine the orientation selectivity of responses in visual cortex measured at a macroscopic scale, and suggest a reinterpretation of a well-established literature on orientation processing in visual cortex.

https://doi.org/10.7554/eLife.37241.001

Introduction

Primary visual cortex (V1) is likely the best studied sensory cortical area, and is a model for understanding broad principles of cortical processing. Orientation in V1 is one of the simplest and best studied cortical sensory features. Orientation is used as a model for understanding more complex feature processing in other cortical areas, and oriented V1-like receptive fields play an important role in successful computational models of vision. Yet the map of orientation in V1 is inadequately understood. At a fine scale, the map shows an orderly periodic structure with pinwheels in hypercolumns, which have a periodicity of about 1 mm in monkeys (Hubel and Wiesel, 1963; Blasdel and Salama, 1986; Grinvald et al., 1986; Das and Gilbert, 1997; Maldonado et al., 1997; Ohki et al., 2006), and about 2 mm in humans (Adams et al., 2007; Yacoub et al., 2008). While the fine-scale structure has been studied extensively, the coarse-scale structure of orientation selectivity is poorly established.

We and others, using fMRI, have described a coarse-scale orientation bias in human V1; each voxel exhibits an orientation preference that depends on the region of space that it represents (Furmanski and Engel, 2000; Sasaki et al., 2006; Mannion et al., 2010; Freeman et al., 2011; Freeman et al., 2013; Larsson et al., 2017). We observed a radial bias most clearly in the peripheral representation of V1: voxels that responded to peripheral locations near the vertical meridian tended to respond most strongly to vertical orientations; voxels along the peripheral horizontal meridian responded most strongly to horizontal orientations; likewise for oblique orientations. This phenomenon had gone mostly unnoticed in single-unit recording studies. fMRI is well-suited to measuring coarse-scale orientation biases because fMRI covers the entire retinotopic map in visual cortex.

The observation of a coarse-scale orientation bias raises two questions. First, what is the relative contribution of coarse- and fine-scale biases to fMRI measurements of orientation selectivity? A number of previous studies have asserted that orientation preferences in fMRI measurements arise primarily from random spatial irregularities in the fine-scale columnar architecture (Boynton, 2005; Haynes and Rees, 2005; Kamitani and Tong, 2005). This conjecture has inspired a large literature on the use of multivariate statistical analyses to exploit these small biases, both to study orientation selectivity in V1 and to study a wide range of other functions throughout the brain. On the other hand, we have argued that the coarse-scale orientation bias is the predominant orientation-selective signal measured with fMRI, and that multivariate decoding analysis methods are successful because of it (Freeman et al., 2011, 2013). The relative contribution of coarse- and fine-scale biases remains hotly debated (Pratte et al., 2016; Alink et al., 2017).

Second, while the neural architecture that gives rise to orientation selectivity is well understood, the neural basis for the coarse-scale orientation bias is unknown. Carlson (2014) proposed that the coarse-scale bias is a byproduct of the edge of the stimulus, referring to it as an ‘edge effect.’ By this account, the coarse-scale bias does not directly reflect enhanced neural responses for particular orientations, but rather arises from properties of the stimulus.

Here, we develop a theoretical account of the edge effect that Carlson (2014) described. We then tested this account empirically. We show that this effect applies not only to stimulus edges but to a much broader class of stimuli. We use the term ‘stimulus vignetting,’ rather than ‘edge effect,’ to emphasize that it is not the edge per se, but rather an interaction between the orientation of the stimulus and a second feature of the display (the aperture or ‘vignette’) that bounds the stimulus. While there are potentially multiple distinct neural mechanisms contributing to coarse-scale orientation biases, we focus here only on characterizing the contribution of stimulus vignetting. We used an image-computable model of V1 activity to generate predictions, which we then tested with fMRI experiments. Our results provide a framework for reinterpreting a wide-range of findings in the visual system.

Results

Coarse-scale bias and stimulus vignetting: Model simulations

We studied the impact of stimulus vignetting on coarse-scale orientation preference maps. The principle underlying stimulus vignetting is that the vignette spreads Fourier energy. The direction and amplitude of this spread is determined by the underlying stimulus spectrum and the aperture geometry, in much the same way that the Fourier transform of a sinusoid windowed by a rectangle consists of the sinusoid frequency convolved with a sinc function. Due to this spectral spread at the aperture edge, neurons with receptive fields straddling the aperture edge respond to a combination of spatial frequency and orientation components, different from neurons with receptive fields far from the edge (Figure 1).

Stimulus vignetting.

(A) Stimulus image (left) and Fourier amplitude (right) for sinusoidal grating that is infinite in extent. Black dots, Fourier amplitude restricted to a single spatial frequency and orientation component. Colored ovals, orientation tuning (polar angle cross-section through the oval) and spatial frequency tuning (radial cross-section) of two hypothetical V1 neurons. Red oval, neuron with broad orientation tuning and narrow spatial frequency tuning. Green oval, neuron with narrow orientation tuning and broad spatial frequency tuning. (B) Sinusoidal grating vignetted by vertical aperture. Light gray dots, spread of Fourier energy because of the aperture. (C) Sinusoidal grating vignetted by horizontal aperture.

https://doi.org/10.7554/eLife.37241.002

We began by simulating the results of our previous fMRI experiment (Freeman et al., 2011) with an image-computable model of V1 (Simoncelli et al., 1992) (see Materials and methods: Theoretical Model). The inputs to the model were identical to the stimuli used by Freeman et al. (2011). Specifically, the stimuli consisted of 16 evenly spaced orientation angles between 0° and 180°. All stimuli were vignetted by a 4° annulus (inner radius, 5°; outer radius, 9°). Both the inner and outer edges of the annulus were blurred with a 1° raised-cosine transition (centered on the inner and outer edges). We measured the model’s responses to each image separately. Each pixel of the model’s output can be thought of as a simulated neuron in the retinotopic map of V1. To simulate an fMRI voxel’s response to the stimuli used in Freeman et al., we summed the model responses across all orientation channels. This procedure yielded a spatial array of simulated voxel responses with different population receptive fields (pRFs). The response of a single simulated fMRI voxel corresponded to a single spatial location on the map of neural responses (e.g., with a pRF straddling the aperture edge). This yielded a set of 16 simulated voxel maps, one for each stimulus orientation (Figure 2). Importantly, we summed across all of the orientation channels. It might be expected, therefore, that the output of the model would not exhibit any orientation tuning. But the simulated fMRI responses do in fact exhibit a clear coarse-scale bias for orientation (Figure 2), replicating previous results (Carlson, 2014). The coarse-scale orientation bias matched Freeman et al. (2011) in that the largest responses were observed for radial orientations (Figure 2).

Simulated neural responses illustrating the impact of stimulus vignetting.

Oriented gratings were vignetted by inner and outer annuli, using stimulus parameters identical to those from Freeman et al. (2011). Next, the gratings were used as input to the model. Model responses were computed separately for vertical (top) and horizontal (bottom) gratings. Finally, model output was computed as the responses to vertical minus responses to horizontal gratings. Model exhibits a preference for horizontal gratings along the horizontal meridian and a preference for vertical gratings along the vertical meridian (i.e. a radial bias).

https://doi.org/10.7554/eLife.37241.003

Having established that stimulus vignetting can produce a radial coarse-scale bias, we next asked whether the computational model predicts the influence of vignetting on responses to novel stimuli. The simulation results suggest that orientation-selective responses measured with fMRI could reflect, at least in part, the shape of the vignette, rather than the underlying orientation tuning of individual neurons. To test this possibility, we generated a novel set of stimuli with vignettes that were orthogonal to one another. If the vignette affects orientation bias, changing the vignette should have a predictable effect on orientation bias. The novel stimuli for this simulation were created as follows. Vertical and horizontal Cartesian gratings (the carrier) were multiplied by a radial or an angular polar grating (the modulator). These compound stimuli were then passed through the model. Subtracting horizontal from vertical carrier grating outputs resulted in an image of orientation bias for the model.

The model predicted that the coarse-scale orientation bias should depend on the modulator. The radial modulator evoked a coarse-scale bias that was radial (Figure 3). In other words, at regions around the vertical meridian there is a greater response for the vertical orientation than for the horizontal orientation (light regions in Figure 3), and vice versa around the horizontal meridian (dark regions in Figure 3). The model predicted that changing the orientation of the modulator should affect the orientation preference to the carrier grating. We found that the angular modulator evoked a coarse-scale bias that was tangential. Thus, at the vertical meridian the horizontal grating yielded higher energy, and at the horizontal meridian the vertical grating evokes higher energy responses.

Model predictions for modulated gratings.

Each of the eight stimuli were created by multiplying a vertical or horizontal grating by a radial or angular modulator. These stimuli were used as input to the model. Model responses were computed as in Figure 2. For radial modulated gratings (top two rows), the model exhibits a radial preference: larger responses to horizontal gratings along the horizontal meridian, larger responses to vertical gratings along the vertical meridian. However, for angular modulated gratings (bottom two rows), the orientation preference is tangential: larger responses for horizontal gratings along the vertical meridian and larger responses for vertical gratings along the horizontal meridian.

https://doi.org/10.7554/eLife.37241.004

The stimuli used in Freeman et al. (2011), and in our simulations in Figure 2, had vignettes with gradual contrast changes, indicating that the impact of vignetting on orientation bias is not dependent on the presence of sharp edges. But is it possible that a more gradual edge would have mitigated vignetting effects? We answered this question by using a sinusoidal modulation that lacked any edges (Figure 3, left). We found similar results for both sinusoidal modulators and modulators with hard edges (i.e. square-wave), as expected, because vignetting arises from any contrast change in the image, rather than an ‘edge’ per se (see Discussion).

Coarse-scale bias and stimulus vignetting: fMRI experiments

We found evidence of stimulus vignetting at both 3T and 7T field strengths, at different spatial resolutions, and for both square wave and sinusoidal modulators, as predicted by the theoretical model. The radial modulator evoked a radial bias, that is, orientation preferences pointing inward toward the fovea (Figure 4, top row). The angular modulator evoked a tangential bias, that is, orientation preferences that were rotated by 90 deg from radial (Figure 4, bottom row). Furthermore, on a voxel-by-voxel basis, preferred orientations were shifted by 90 deg across modulators (Figure 5). Comparing model predictions to data from individual voxels, we observed a clear correspondence between predicted and measured orientation preference (Figure 6). Circular correlations between predicted and measured orientation preferences were 0.26 and 0.20 for radial and angular modulators, respectively (p<0.0001 for both). Note that many sources of noise in the measurements could impact this analysis: in estimating the location and size of each voxel’s pRF, in measuring each voxel’s orientation preference, and in coregistering each voxel’s response across multiple fMRI sessions. Moreover, the model we use is highly simplified; for example, it does not take into account changes in spatial frequency tuning at greater eccentricities. Yet, despite the multiple sources of noise and the simplified assumptions of the model, the correspondence between the model’s prediction and the empirical measurements is highly statistically significant.

Figure 4 with 1 supplement see all
fMRI measurements of orientation bias depend on stimulus vignetting.

(A) Conventional resolution, 3T. Top: Responses to phase-encoded oriented gratings multiplied by a static radial modulator (shown in inset). The oriented grating cycled through 16 steps of orientation ranging from 0° to 180° every 24 s. The radial modulator was constant during the entire experiment. Map thresholded at coherence of 0.2. Hue indicates phase of the best fitting sinusoid. White lines indicate V1/V2 boundaries. Bottom: Responses to the same oriented grating as in A, but here the grating is multiplied by an angular modulator. As predicted by the model, the radial modulator gave rise to a radial orientation bias, while the angular modulator gave rise to a tangential orientation bias. (B) High-resolution, high field strength (7T) measurements of orientation preference for radial and angular modulators. Stimuli and conventions same as for A, except the modulators were radial and angular sinusoids.

https://doi.org/10.7554/eLife.37241.005
Orientation bias depends on the modulator.

Orientation bias for V1 voxels pooled across observers for the angular modulator (y-axis) and radial modulator (x-axis).

https://doi.org/10.7554/eLife.37241.007
Combining the model with pRF estimates enables reliable predictions of single-voxel orientation preference.

(A) pRF sampling method. (1) Each voxel’s pRF was estimated based on an independent pRF-mapping scanning session. (2) Model output was sampled by the voxel pRF, resulting in an estimated preferred orientation for each modulator. (3) Estimated preferred orientation was compared to measured orientation preference, for each modulator. (B) Estimated orientation preference plotted against measured orientation preference. Sampling the model output with estimated pRFs yielded accurate predictions for each voxel’s orientation preference for each of the two modulators, indicating that the model of stimulus vignetting provides a good account for the fMRI measurements of orientation preference.

https://doi.org/10.7554/eLife.37241.008

We hypothesized that if stimulus vignetting accounts for the radial bias, and changing the direction of the modulator flips voxels’ orientation preferences by 90 deg, then the population pattern of activity should also shift by 90 deg. We tested this hypothesis by training a classifier on fMRI responses to the radial modulator, and testing how accurately it decodes responses to the angular modulator, and vice versa. We compared between-modulator decoding accuracy to accuracy obtained from training and testing with data from the same modulator (within-modulator). Within-modulator decoding yielded high accuracies (Figure 7A, left, p<0.001, tested with a non-parametric permutation test). Across-modulator decoding accuracy was below chance (Figure 7A, middle, p<0.0001). These results indicate that the population response to oriented gratings was dependent on the modulator — responses to an oriented grating multiplied by one modulator (e.g. radial) were entirely different from responses to the same oriented grating when multiplied by the other modulator (e.g., by the angular modulator).

Population responses reflect the modulator, not stimulus orientation.

(A) fMRI decoding. Within-modulator decoding: classifier was trained and tested on data from each modulator separately, in a leave-one-run-out cross-validation approach. Across modulator decoding: classifier was trained on data from one modulator and tested on data from the other modulator (and vice-versa). Across-modulator decoding after shifting: data for one modulator was shifted by 90°, then the classifier was trained on that modulator and tested on the other (and vice-versa). (B) Multidimensional scaling. Top: population responses to all orientations for both modulators (solid lines, radial modulator; dotted lines, angular modulator) are plotted in 2D, such that distances between conditions reflect how different (i.e., dissimilarity = 1–r, where r is the correlation coefficient) the responses are. For the original data, orthogonal orientations from the two modulators have similar responses and are plotted near each other, while identical orientations taken from the two modulators evoked very different responses, and lay distant from each other. Bottom: after shifting the labels for the angular modulator by 90 deg, the two datasets align well, and similar orientations correspond to similar population responses regardless of the modulator.

https://doi.org/10.7554/eLife.37241.009

Does changing the modulator (between radial and angular) shift the population response pattern by 90 deg, as predicted by the model? To test this possibility, we repeated the between-modulator decoding analysis described above, but shifted the orientation labels by 90°. After shifting the orientation labels, decoding accuracy returned to levels significantly above chance (Figure 7A, right, p<0.0001), only slightly lower than within-modulator decoding accuracy on the unshifted data. The difference between within-modulator decoding accuracy and between-modulator decoding accuracy after shifting labels was small but statistically significant (p=0.046). Note that the two modulators share inner and outer radial edges. Thus, not all voxels are expected to shift by 90 deg. These common edges may be the cause of the slightly lower decoding accuracy across modulators. However, we cannot rule out the possibility that other sources of orientation-selective information not associated with the vignette, are present in the data.

We performed a complementary analysis, based on multidimensional scaling, to visualize the responses for both modulators in a single 2D plot (Figure 7B). Each line in Figure 7B represents a population response to a single stimulus orientation. The color and the line orientation in the plot both indicate the orientation of the stimulus. The distances between the lines reflect the dissimilarity between the corresponding population responses: similar responses are near to each other, and dissimilar responses are further apart. For each modulator, we found the responses to different orientations to be organized continuously in a ring. The two rings, corresponding to the two modulators, are misaligned with each other, reflecting the change in the population responses resulting from the change in modulator (Figure 7B). Specifically, responses to identical orientations (i.e. identical colored lines) are distant, while responses to dissimilar orientations are proximal. However, after rotating the labels for responses to the angular modulator by 90 deg, the two sets of responses are well aligned. This demonstrates once again that changing modulators completely flips voxels’ orientation bias, and this is reflected throughout the V1 retinotopic map.

Discussion

Our results show that stimulus vignetting strongly affects voxel orientation bias. This suggests that results from many fMRI studies attempting to measure neural orientation tuning properties may actually reflect stimulus vignetting. Since stimulus vignetting masquerades as orientation selectivity amongst voxels that correspond to the stimulus edge, and because edges tend to be continuous over a large spatial extent, vignetting will generally result in a coarse-scale map of orientation bias. In many fMRI studies, gratings are presented within a circular annulus (Haynes and Rees, 2005; Kamitani and Tong, 2005; Harrison and Tong, 2009; Freeman et al., 2011; Sun et al., 2013; Larsson et al., 2017; Wardle et al., 2017), and in these cases the model predicts a radial coarse-scale bias. However, in some cases, other stimulus vignetting exists, resulting in more subtle coarse-scale patterns of orientation bias (Swisher et al., 2010; Larsson et al., 2017; Sengupta et al., 2017). One study (Alink et al., 2017) used inner and outer circular annuli, but added additional angular edges, the result of which should be a combination of radial and tangential biases. Indeed, this study reported that voxels had a mixed pattern of selectivity, with a considerable number of voxels reliably preferring tangential gratings, and other voxels reliably favoring radial orientations.

Stimulus vignetting is a general issue of concern in visual neuroscience (Carter and Henning, 1971; Larsson et al., 2017; Tang et al., 2018). According to the theoretical framework that we have applied here, whenever a neuron’s receptive field overlaps a stimulus edge or a change in contrast, stimulus vignetting will spread the Fourier power and affect the neuron’s response. As a result, such a neuron may exhibit ostensible orientation tuning even when it is not orientation selective. Furthermore, even if a neuron is truly orientation selective, measurements of its orientation tuning may be confounded by the aperture shape.

Vignetting may produce ostensible selectivity to second-order features, including vignettes themselves. For example, consider the modulated gratings that we used in the current study. A V1 neuron with a receptive field near the vertical meridian (i.e., near the V1/V2 border) might respond more to a radial modulated vertical grating than to an angular modulated vertical grating simply as a result of vignetting. This could be misinterpreted as sensitivity to second-order orientation contrast (Zhou and Baker, 1993; Mareschal and Baker, 1998; Song and Baker, 2007), when in fact the neuron is sensitive only to first-order orientation. There are no so-called ‘second-order filters’ in our model. Yet it exhibits orientation-selective biases for 2nd-order modulation because of vignetting. Similarly, vignetting might account for ostensible selectivity to orientation of illusory contours and second-order patterns, in both single-cell (Grosof et al., 1993) and fMRI studies (Larsson et al., 2006; Montaser-Kouhsari et al., 2007).

Vignetting may even produce ostensible orientation selectivity in neurons that are not orientation selective at all. A major question concerning the neural origin of orientation selectivity is whether and to what degree lateral geniculate nucleus (LGN) responses are tuned to orientation (Ling et al., 2015; Sun et al., 2004; Sun et al., 2016). Recently, one group addressed this question using fMRI. They found that orientation can be decoded from LGN, suggesting that thalamic responses are tuned to orientation (Ling et al., 2015), perhaps inheriting their selectivity from the retina (Schall et al., 1986). Another group, using calcium imaging to measure bouton responses in mice, found extensive orientation tuning as well (Sun et al., 2016). But stimulus vignetting may have played a critical role in both of these studies. In the former study (Ling et al., 2015), gratings were presented within a circular annulus, likely yielding a radial bias due to stimulus vignetting. In the latter study (Sun et al., 2016), the grating covered the entire visual display (75 deg x 75 deg), which is nearly the entire extent of the mouse monocular visual field. Hence, it might seem that vignetting would not have been a dominant source of orientation selectivity in this experiment. However, mouse neuronal receptive fields are large, at least an order of magnitude larger than in primates (Van den Bergh et al., 2010; Tang et al., 2016). A significant portion of neurons recorded by Sun et al. (2016) may have therefore had receptive fields that overlapped the edges of the screen, which would have resulted in vignetting. These and many other related studies may not have been measuring orientation selectivity at all, but rather activity reflecting stimulus vignetting.

Do voxel responses contain any orientation preferences that are not the result of stimulus vignetting? We found no evidence for any other orientation information present in the data. Based on the current study we cannot rule out the possibility of other factors affecting fMRI orientation preferences, at both coarse and fine spatial scales. For example, fine-scale signals originating from a columnar organization may, theoretically, affect fMRI signals (Boynton, 2005). But these columnar signals may be entirely eclipsed by stimulus vignetting. Similarly, coarse-scale signals such as a physiological radial bias may indeed exist, but be relatively small, and hence also eclipsed by stimulus vignetting. Our previous results, describing a vertical bias near the fovea (Freeman et al., 2013), in addition to the radial bias in the periphery (Freeman et al., 2011), lend support to the possibility that there are multiple sources of coarse-scale bias.

Do fMRI voxels reflect any fine-scale orientation signals originating from a columnar organization, in addition to an orientation bias resulting from vignetting? If each voxel reflects the activity of a large number of neurons, signals from multiple orientation columns will be summed within a voxel, thus attenuating any columnar signal. If this is the case, decreasing the voxel size should lower the number of different columns summed within each voxel, and amplify columnar orientation selectivity. On the other hand, orientation bias resulting from vignetting should not depend on scanning resolution. Nearby neurons respond to nearby locations in the visual field, and therefore will respond similarly to the vignette. Thus, increasing the acquisition resolution will augment columnar signals but not vignetting signals. Our results at 2 mm voxels demonstrate primarily vignetting with no evidence of a columnar signal. Increasing the acquisition resolution to 0.9 mm did not alter this result. From this we conclude that the coarse-scale bias due to vignetting is the predominant source of orientation information measured with fMRI at scanning resolutions with voxel sizes of 1 mm or larger.

We are no longer arguing about whether orientation decoding is dominated by coarse-scale orientation biases; we already know that it is. An extreme method we previously used for testing whether fine-scale information contributes to fMRI orientation decoding involved shifting the acquired slices by half a voxel (1 mm), and then attempting to decode the grating orientation (Freeman et al., 2013). If successful decoding depends on fine-scale information, shifting the slices should eradicate any decoding information. In fact, decoding accuracy was unaffected (statistically indistinguishable) by shifting the slices. However, the finding that a coarse-scale bias is the source of orientation decoding remains controversial, and several recent studies have attempted to disprove it (Alink et al., 2013; Pratte et al., 2016; Alink et al., 2017), in part, we believe, because the notion that fMRI is sensitive to fine-scale neural activity is highly attractive. We have no doubt that there are orientation columns in human V1. But we also have no doubt that orientation-decoding with fMRI at conventional resolutions is dominated by coarse-scale orientation bias, not the fine-scale (columnar) architecture for orientation selectivity. Otherwise, decoding would have suffered from shifting the slices by half the hypercolumn width. Rather than continuing to debate about whether fMRI decoding is dominated by coarse- or fine-scale information, the current study characterizes one of the sources of the coarse-scale bias, the implications of which are much broader than the technical details of orientation-decoding with fMRI (see above).

Leaving aside this debate about whether orientation decoding with fMRI depends on coarse- vs. fine-scale information, it is important to remember that the contribution of many decoding studies remains, irrespective of the source of decodable information. For example, a primary contribution of Kamitani and Tong (2005) regards feature-based attention: they showed that attending to one or the other orientation changes the activity in V1 in such a way that the attended orientation can be decoded. That result holds regardless of whether the decoding depends on coarse- vs. fine-scale. Similarly, Haynes and Rees (2005) showed that V1 activity reflects properties of invisible stimuli, using decoding. Likewise for many studies that used decoding, forward modeling, and other MVPA techniques to reveal properties of cortical processing. Most, if not all, of these findings do not depend on the source of decodable information.

Stimulus vignetting has previously been referred to as an ‘edge effect’, and we demonstrated the intuition of stimulus vignetting using examples of hard edges (Figure 1). Several studies have used gratings with smoothed edges, and some have smoothed the edges in an attempt to eliminate any effects related to the edge (Freeman et al., 2011; Warren et al., 2014). According to the model, however, vignetting does not occur exclusively in the vicinity of edges. Rather, a contrast change, even a gradual change, is all that is necessary to produce stimulus vignetting. The vignette causes the Fourier spectrum to splatter (Figure 1). Smoothing the stimulus edge only acts to spatially spread this effect across a larger portion of the visual field. Indeed, we found that both square-wave modulators and sinusoidal modulators evoked similar levels of vignetting, empirically confirming the model’s prediction.

Other factors, in addition to stimulus vignetting, may contribute to the coarse-scale bias. First, the coarse-scale orientation bias may reflect, in addition to stimulus vignetting, a neuronal ‘gain map’ (Philips and Chakravarthy, 2016). Neurons in visual cortex may have different response gains depending on their preferred orientations and receptive field locations. For example, the responses of a population of neurons tuned for vertical might be highest for those neurons with receptive fields near the vertical meridian and smallest for those along the horizontal meridian, and vice versa for neurons tuned for horizontal, conferring a cardinal bias. A gain map could be instantiated in visual cortex through a number of different physiological mechanisms. For example, in a population of neurons along the horizontal meridian, a gain map might be instantiated by (1) a larger proportion of neurons that prefer horizontal orientations, or (2) higher firing rates for the sub-population of neurons that prefer horizontal orientations, or (3) a wider tuning bandwidth for the sub-population of neurons that prefer orientations close to vertical. Each of these three mechanisms would give rise to a larger population response for horizontal gratings in fMRI voxels that lay along the horizontal meridian.

Second, the coarse-scale orientation bias might reflect ‘asymmetric surround suppression’, an imbalance in suppression between different parts of the region surrounding a neuron’s receptive field. The responsiveness of a V1 neuron is suppressed by stimuli outside its classical RF (Sceniak et al., 1999; Sceniak et al., 2001; Cavanaugh et al., 2002a; Cavanaugh et al., 2002b; Zenger-Landolt and Heeger, 2003). Suppression is stronger for neurons with RFs near the center of a stimulus aperture, less so for neurons with RFs near the aperture edge, because there is less stimulation surrounding those RFs. For many neurons in V1, surround suppression is asymmetric, such that there is more suppression at the ‘ends’ of the receptive field (i.e., along the preferred orientation) than the ‘sides’ (Larsson et al., 2017; Cavanaugh et al., 2002a; Cavanaugh et al., 2002b; Tanaka and Ohzawa, 2009; Hallum and Movshon, 2014). Because of this asymmetry, the release from suppression may be greater for aperture edges that are orthogonal to each neuron’s preferred orientation, such that a circular aperture centered at fixation will confer a radial bias.

Regardless of whether a gain map and/or asymmetric surround suppression contribute to coarse-scale orientation bias, stimulus vignetting shows a strong and robust effect on measured orientation preference. Orientation is used as a model for understanding more complex feature processing in other cortical areas, and oriented V1-like receptive fields play an important role in successful computational models of vision. Yet, the computations that give rise to orientation tuning in V1 remain inadequately understood. Our results provide a framework for reinterpreting a wide-range of findings on orientation selectivity.

Materials and methods

Theoretical model

The image-computable model is based on the steerable pyramid (Simoncelli et al., 1992), a subband image transform that decomposes an image into orientation and spatial frequency channels (Figure 1). The pyramid simulates the responses of a large number of linear receptive fields (RFs), each of which computes a weighted sum of the stimulus image; the weights determine the orientation and spatial frequency tuning. RFs with the same orientation and spatial-frequency tuning, but shifted to different locations, are called a ‘channel’. In the model, the number of spatial frequency channels, orientation channels, and orientation bandwidth are model parameters that can each be chosen flexibly. We used an instantiation of the model with six orientations, which resulted in a bandwidth comparable to empirical tuning curves measured in primate electrophysiological recordings (Ringach et al., 2002). Using four or more bands with correspondingly broader or narrower tuning curves yielded similar results supporting the same conclusions. The number of spatial frequency channels was determined by the size of the input images and the spatial frequency bandwidth. We used images that were 768 × 1024 pixels, and a spatial frequency bandwidth of 0.5 octaves, and so the model had 15 different spatial frequency channels. The RFs cover all orientations and spatial frequencies evenly (i.e., the sum of the squares of the tuning curves is exactly equal to one for all orientations and spatial frequencies). For each orientation and spatial frequency, the pyramid includes RFs with two different phases, analogous to odd- and even-phase Gabor filters. The sum of the squares of the responses of two such RFs computes what has been called an ‘energy’ response (Adelson and Bergen, 1985; Heeger, 1992) because it depends on the local spectral energy within a spatial region of the stimulus, for a particular orientation and spatial frequency. The energy responses are also arranged in channels. The linear RFs in the model are hypothesized to be a basis set of orientation and spatial frequency tuning curves of V1 neurons; any invertible linear transform of the basis set can be substituted (Freeman and Adelson, 1991; Simoncelli et al., 1992; Simoncelli, 1993) and different such transforms can be substituted at each spatial location, consistent with the diversity in the tuning properties of V1 neurons, without changing the nature of the representation. The model was designed so that the filters evenly tile retinotopic space, orientation, and spatial frequency to avoid any possibility of the kind of artifact that was suggested to underlie Carlson (2014) simulation results (Clifford and Mannion, 2015).

The strength of the model is its simplicity. Our goal is not to provide an accurate model of V1 activity, but rather to provide a platform for assessing vignetting. We purposely kept the model simple so that we could be certain that vignetting is indeed the source of orientation bias in the model. Ostensibly interesting features of V1 responses (e.g. second-order contrast sensitivity) might actually be a result of vignetting. If we were to include such features in the model, any orientation bias could be a combined result of vignetting and those features. Keeping the model simple eliminates that possibility.

We used the theoretical model to predict fMRI responses to a wide range of stimuli. We then chose stimuli that were predicted to have opposite patterns of orientation bias when measured with fMRI. Predicted responses were calculated by finding the scale with maximal responses, summing the energy responses for the channels at that scale, and averaging across stimulus grating phases (See: fMRI experiments, Stimuli).

fMRI experiments

fMRI experiments were conducted at two sites. Experiments at 3T were conducted at NYU Center for Brain Imaging. Experiments at 7T were conducted at Functional Magnetic Resonance Imaging Core Facility at NIH. Methods were similar between the two sites except where noted.

Stimuli

Stimuli consisted of two gratings (a carrier and a modulator) multiplied by one another (Figure 3). The carrier grating consisted of a large, oriented sinusoidal Cartesian grating (spatial frequency, NYU: 0.5 cycles per degree; NIH: 1.4 cycles per degree) presented within an annulus (NYU: inner radius, 0.5°; outer radius 9.5°; NIH: inner radius 0.75°; outer radius 9°). The spatial phase of the carrier grating was randomized every 250 ms from a predefined set of 16 phases uniformly distributed between 0 and 2π. The orientation of the carrier grating cycled through 16 evenly- spaced angles between 0° and 180° (1.5 s per orientation). The carrier grating completed 10.5 cycles in each run. Each cycle was 24 s long, producing a run length of 4 min, 12 s. The orientation of the carrier grating cycled clockwise in half of the runs and counter-clockwise in the other half.

The modulator grating was a polar-transformed grating. In the ‘square wave modulator’ experiments (at NYU and NIH), the modulator grating was square, with hard edges. In the ‘sinusoidal modulator’ grating experiment (NIH), the modulator did not have any edges. On half the runs the modulator had a radial orientation, and an angular orientation in the other half. The modulator grating produced either a set of rings emanating from the fovea (radial orientation) or a set of inward-pointing wedges encircling the fovea (angular orientation). In the NIH experiments, the rings were scaled with eccentricity. On half the runs, the modulator grating was sine phase and cosine phase in the other half. We tested every combination of clockwise/counter-clockwise cycling carrier gratings, radial/angular modulator gratings, and sine/cosine phase modulator gratings. This produced a total of 8 conditions. Each of the eight conditions was tested in two independent runs, producing a total of 16 runs per scanning session.

Because the only stimulus feature that varied within each run was the orientation of the carrier grating, any fMRI activity measured within each run could be attributed to either the orientation of the carrier grating, or an interaction between the orientation of the carrier grating and the static modulator grating. The modulator itself did not evoke any modulation in the fMRI responses because it did not change during the course of a scanning run.

Observers

Thirteen observers (eight females, aged 22–27 years) with normal or corrected-to-normal vision participated in the study. Observers provided written informed consent.

The consent and experimental protocol were in compliance with the safety guidelines for MRI research, and were approved by both the University Committee on Activities involving Human Subjects at New York University, and the Institutional Review Board at National Institutes of Health.

Each observer participated in multiple scanning sessions. Five observers (O1–O5) each participated in one session of the ‘square wave modulator’ experiment on the 3T scanner at NYU. Seven observers (O6–O11, O13) each participated in two sessions at NIH: one session for the ‘square wave modulator’ experiment and one session for the ‘sine wave modulator’ experiment. Data from a single session were discarded due to severe ghosting in the BOLD images. One observer (O12) did not participate in the ‘sine wave modulator’ experiment. In addition, each of the thirteen observers participated in one session to obtain a set of high-resolution anatomical volumes, used for cortical segmentation, and an additional session for retinotopic mapping.

Orientation mapping experiments

Stimuli were generated using Matlab (MathWorks, MA) and MGL (Gardner et al., 2018b) on a Macintosh computer. Stimuli were displayed via an LCD projector (3T scanner: Eiki LC-XG250; resolution: 1024 × 768 pixels; refresh rate: 60 Hz. 7T scanner: PLUS U2-1200; resolution: 800 × 600 pixels; refresh rate: 60 Hz) onto a back-projection screen in the bore of the magnet. Observers viewed the display through an angled mirror at a viewing distance of approximately 58 cm (field of view in the 3T at NYU: 31.6 deg × 23.7 deg; field of view in the 7T at NIH: 20.5 deg × 16.1 deg).

The novel stimuli consisted of two gratings (a carrier and a modulator) multiplied by one another (Figure 3). The carrier grating consisted of a large, oriented sinusoidal Cartesian grating presented within an annulus. The spatial phase of the carrier grating was randomized every 250 ms from a predefined set of 16 phases uniformly distributed between 0 and 2π. The orientation of the carrier grating cycled through 16 evenly-spaced angles between 0° and 180° (1.5 s per orientation). The carrier grating completed 10.5 cycles in each run. Each cycle was 24 s long, producing a run length of 4 min, 12 s. The orientation of the carrier grating cycled clockwise in half of the runs and counter-clockwise in the other half.

Behavioral task

Throughout each run, observers continuously performed a demanding two-interval, forced-choice task to maintain a consistent behavioral state and stable fixation, and to divert attention from the main experimental stimuli. In each trial of the task, the fixation cross (a 0.4 deg crosshair) dimmed twice (for 400 ms at a time), and the observer indicated with a button press the interval (1 or 2) in which it was darker. The observer had 1 s to respond and received feedback through a change in the color of the fixation cross (correct green, incorrect red). Each contrast decrement and response period was preceded by a 800 ms interval, such that each trial lasted 4.2 s. The fixation task was out of phase with the main experimental stimulus presentation. Contrast decrements were presented using an adaptive, 1-up-2-down staircase procedure (Levitt, 1971) to maintain performance at approximately 70% correct.

MRI data were acquired on two different scanners at two different sites. At NYU, data were acquired for observers O1-O5 on Siemens (Erlangen, Germany) 3T Allegra head-only scanner using a transmit head coil (NM-011, Nova Medical, Wakefield, MA), and an eight-channel phased-array surface receive coil (NMSC-071, Nova Medical). Functional imaging was conducted with 24 slices oriented perpendicular to the calcarine sulcus and positioned with the most posterior slice at the occipital pole (TR: 1500 ms; TE: 30 ms; FA: 72°; voxel size: 2 × 2 × 2.5 mm; grid size: 104 × 80 voxels).

At NIH, data were acquired from observers O6-O13 on a research-dedicated Siemens 7T Magnetom scanner using a 32-channel head coil, located in the Clinical Research Center on the National Institutes of Health campus (Bethesda, MD). Functional imaging was conducted with 54 slices oriented perpendicular to the calcarine sulcus covering the posterior half of the brain (TR: 1500 ms; TE 23 ms; FA: 55°; voxel size: 1.2 × 1.2 × 1.2 mm or 0.9 × 0.9 × 0.9 mm with 0% or 10% gap between slices, respectively; grid size: 160 × 160 voxels. Multiband factor 2 or 3, GRAPPA/iPAT factor 3).

For each observer, a high-resolution anatomy of the entire brain was acquired by co-registering and averaging between 2 and 8 T1-weighted anatomical volumes (magnetization-prepared rapid-acquisition gradient echo, or MPRAGE; TR: 2500 ms; TE: 3.93 ms; FA: 8°; voxel size: 1 × 1 × 1 mm; grid size: 256 × 256 voxels). The averaged anatomical volume was used for co-registration across scanning sessions and for gray-matter segmentation and cortical flattening. Functional scans were acquired using T2*-weighted, gradient recalled echo-planar imaging to measure blood oxygen level-dependent (BOLD) changes in image intensity (Ogawa et al., 1990). At NYU, an MPRAGE anatomical volume with the same slice prescription as the functional images (‘inplane’) was also acquired at the beginning of each scanning session (TR: 1530 ms; TE: 3.8 ms; FA: 8°; voxel size: 1 × 1 × 2 mm with 0.5 mm gap between slices; grid size: 256 × 160 voxels). The inplane anatomical was aligned to the high-resolution anatomical volume using a robust image registration algorithm (Nestares and Heeger, 2000).

At NIH, prior to the first experimental functional run of each session, 30 volumes were acquired with identical scanning parameters and slice prescription as the subsequent functional runs, except for the phase encoding direction which was reversed. This scan was used for correcting spatial distortions (see next section).

fMRI time series preprocessing

The single reverse phase-encoded run was used to estimate the susceptibility-induced off-resonance field using a method similar to that described in (Andersson et al., 2003) as implemented in FSL (Smith et al., 2004). This estimate was then used to correct the spatial distortions in each subsequent run in the session. Data from the first half cycle (eight volumes) at the beginning of each functional run were discarded to minimize the effect of transient magnetic saturation and to allow hemodynamic response to reach steady state: the first half cycle. Functional data were compensated for head movement within and across runs (Nestares and Heeger, 2000), linearly detrended, and high-pass filtered (cutoff: 0.01 Hz) to remove low-frequency noise and drift (Smith et al., 1999). The time series for each voxel was divided by its mean to convert from arbitrary intensity units to percent change in image intensity. Time series data from each run were shifted back in time by three volumes (4.5 s) to compensate approximately for hemodynamic lag. Time series for the clockwise runs were time reversed to match the sequence of the counterclockwise runs. Time series were then averaged within modulator type (the matrix of 8 conditions: cosine and sine modulator crossed with clockwise and counterclockwise orientation). Each individual voxel’s time-course was fitted to a cosine, and the orientation corresponding to the phase of the cosine was assigned as that voxel’s preferred orientation. It should be noted that shifting, inverting and averaging minimizes differences in hemodynamic responses between voxels, and results in a smooth time series that fits well to a cosine. Without this stage, the time series would have less Fourier power at the fundamental frequency and much of the power would instead shift to harmonic frequencies (see (Pratte et al., 2016) for examples of these effects).

Retinotopic maps

Retinotopy was measured using non-periodic traveling bar stimuli and analyzed using the population receptive field (pRF) method (Dumoulin and Wandell, 2008). Bars were 3 deg wide and traversed the field of view in sweeps lasting 24 s. Eight different bar configurations (4 orientations and two traversal directions) were presented. The pRF of each voxel was estimated using standard fitting procedures (Dumoulin and Wandell, 2008), implemented in Matlab using mrTools (Gardner et al., 2018a). Visual area boundaries were drawn by hand on the flat maps, following published conventions (Engel et al., 1997; Larsson et al., 2017; Wandell et al., 2007).

pRF sampling analysis

To test the model’s predictions for the responses of individual fMRI voxels, we estimated pRFs and sampled the model’s output at the corresponding region of the visual field. Specifically, each voxel’s Gaussian pRF was used to weight the model output for each grating orientation, averaged across grating phase and modulator phase, resulting in a single predicted response value for each orientation. We then fitted a cosine to the responses, with the phase reflecting the voxel’s predicted preferred orientation. This was done separately for each of the two modulators. Finally, we compared the predicted preferred orientation to the measured preferred orientation (see Figure 6, top).

fMRI MVPA analysis

Classification was performed on fMRI time series, in a leave-one-run-out manner, with a naive Bayes classifier (using Matlab function ‘classify’ with ‘diagLinear’ option). First, for each voxel, we averaged across all 10 cycles within a run, yielding 16 time points per voxel, which reflected the amplitude of the response to each stimulus orientation. We refer to these 16 vectors across voxels as population responses, a single response vector per time point, or equivalently, per orientation.

For within-modulator decoding, the classifier was trained on data (response vectors and corresponding orientation labels) from seven runs, and tested on the 8th run (leave-one-run-out decoding). If there were consistent differences in the responses to different orientations the classifier should decode correctly at an accuracy rate that is above chance level. We then cross-validated, by repeating this procedure eight times, each time leaving out a different run for testing. This was repeated for both modulators, and results were averaged across them.

For across-modulator decoding, the classifier was trained on data from one modulator, then tested on data from the other modulator. We then swapped the training and testing datasets and repeated the analysis. For shifted across-modulator decoding, we first shifted the orientation labels for the angular modulator by 90 deg, then repeated the across-modulator decoding procedure, and averaged the accuracy. Statistical significance of decoding accuracy was determined by randomly permuting the phases of the Fourier transform of the raw time-course 10,000 times and comparing the mean accuracy across subjects to the mean accuracy without permuting. Decoding accuracy X was deemed significant at p<a if the fraction of permutations that yielded decoding accuracies higher than X was smaller than a.

While decoding enabled us to measure orientation information present in population responses, it provided only an indirect view of the shift the responses undergo, as a result of changing the modulator. To visualize the responses in a more direct fashion, we used representational similarity analysis. Each condition (i.e., 16 orientations for each of 2 modulators), represented by a population response vector averaged across cycles and runs, was correlated with all other conditions, yielding a correlation matrix. The matrix was transformed into a dissimilarity matrix by subtracting from 1 (i.e., dissimilarity = 1-r, where r is the correlation coefficient). We then averaged across subjects and used multidimensional scaling (Matlab function ‘mdscale’) to reduce the dimensionality to 2. Results are plotted both with the original orientation labels, and after shifting the angular modulator’s orientation labels by 90 deg.

Data and code sharing

We make publicly available all of the fMRI data reported here, as well as open-source software for implementing the theoretic model of V1, generating stimuli for the fMRI experiments, and analyzing the fMRI data (Merriam and Roth, 2018), at https://doi.org/10.17605/OSF.IO/TBJRF.

References

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
    The design and use of steerable filters
    1. WT Freeman
    2. EH Adelson
    (1991)
    IEEE Transactions on Pattern Analysis and Machine Intelligence 13:891–906.
    https://doi.org/10.1109/34.93808
  20. 20
  21. 21
    mrTools: Analysis and visualization package for functional magnetic resonance imaging data
    1. JL Gardner
    2. EP Merriam
    3. D Schluppeck
    4. J Besle
    5. DJ Heeger
    (2018)
    mrTools: Analysis and visualization package for functional magnetic resonance imaging data, 4.7.
  22. 22
    MGL: Visual psychophysics stimuli and experimental design package
    1. JL Gardner
    2. EP Merriam
    3. D Schluppeck
    4. J Larsson
    (2018)
    MGL: Visual psychophysics stimuli and experimental design package, 2.0.
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
    Transformed up-down methods in psychoacoustics
    1. H Levitt
    (1971)
    The Journal of the Acoustical Society of America 49:467–477.
    https://doi.org/10.1121/1.1912375
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
    Data and code for “Stimulus vignetting and orientation selectivity in human visual cortex.
    1. EP Merriam
    2. ZN Roth
    (2018)
    Data and code for “Stimulus vignetting and orientation selectivity in human visual cortex..
  40. 40
  41. 41
  42. 42
  43. 43
  44. 44
  45. 45
  46. 46
  47. 47
  48. 48
  49. 49
  50. 50
  51. 51
  52. 52
  53. 53
    Distributed Representation and Analysis of Visual Motion
    1. EP Simoncelli
    (1993)
    Massachusetts Institute of Technology.
  54. 54
  55. 55
  56. 56
  57. 57
  58. 58
  59. 59
  60. 60
  61. 61
  62. 62
  63. 63
  64. 64
  65. 65
  66. 66
  67. 67
  68. 68
  69. 69
  70. 70
  71. 71

Decision letter

  1. Floris P de Lange
    Reviewing Editor; Donders Institute for Brain, Cognition and Behaviour, Netherlands
  2. Sabine Kastner
    Senior Editor; Princeton University, United States

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your article "Stimulus vignetting and orientation selectivity in human visual cortex" for consideration by eLife. Your article has been reviewed by three peer reviewers, including Floris P de Lange as the Reviewing Editor and Reviewer #1, and the evaluation has been overseen by David Van Essen as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: Kendrick N Kay (Reviewer #2); Nikolaus Kriegeskorte (Reviewer #3).

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

Summary:

The manuscript uses computational modeling and experimental measurements to make a compelling case that stimulus edge effects (or 'vignetting') may be a substantial source of apparent orientation tuning as measured using standard-resolution fMRI. The work is technically solid and carefully presented, and the topic will be of significant interest to the fMRI community (given the widespread use of multivoxel pattern analysis).

Essential revisions:

I have provided a summary of essential revisions. You will find more details, as well as additional points that need to be addressed, in the original reviews (appended below).

1) Tone down claims of novelty:

-The paper claims to introduce a novel idea that requires reinterpretation of a large literature. The claim of novelty is unjustified. Vignetting was discovered by Carlson et al., 2014 and in Wardle et al., 2017, Carlson's group showed that it may be one, but not the only contributing factor enabling orientation decoding. Carlson et al. deserve clearer credit throughout. See reviewer #2 point 1, and reviewer #3.

2) Provide a more nuanced coverage of the literature. E.g., the conjecture that the orientation preferences in fMRI measurements arise primarily from random spatial irregularities in the fine-scale columnar architecture doesn't seem 'leading' anymore in 2018, and more nuanced positions have been articulated since. See reviewer #1 point 1 and reviewer #3.

3) Discuss the broader implications of the study: the authors claim that the study has wide implications for many studies that used decoding of oriented gratings. But it is left unspecified what those implications are. Could the authors be more specific? For example, how should we reinterpret Kamitani and Tong, 2005 or Haynes and Rees, 2005? What wrong conclusions have been drawn, if we accept the notion that stimulus vignetting is the source of orientation decoding?

The significance of the present work might be further emphasized by relating it more broadly to the general approach of MVPA (i.e. using linear classifiers to decode activity). I believe the larger lesson, highlighted by the present work, is that even seemingly simple properties are in fact hard to isolate experimentally and that powerful approaches like classification can pick up on aspects of the data that might not be what the experimenter intended. See reviewer #1 point 4, reviewer #2 point 2.

4) Address/discuss whether decoding is still possible in the absence of vignetting effects, or is solely dependent on vignetting. See reviewer #1 point 2, 3; reviewer #3.

Reviewer #1:

The authors test the hypothesis that the orientation selectivity of responses in visual cortex, measured at a macroscopic scale, is caused by 'vignetting', i.e. second order changes in luminance caused by the aperture within which an oriented grating is presented. Using a computational model of neural responses in V1, they demonstrate the impact of vignetting on the model response, and confirm predictions made by the model in human fMRI measurements.

I find this a compelling manuscript, with a clear question that is of interest to many researchers. The results are expertly analyzed and clear cut, showing that stimulus vignetting may indeed be a major contributor to the orientation selectivity measured with fMRI. I have some comments related to the coverage of the literature, and a query about the implications of the study for the field.

1) Introduction section: "A leading conjecture is that the orientation preferences in fMRI measurements arise primarily from random spatial irregularities in the fine-scale columnar architecture (Boynton, 2005; Haynes and Rees, 2005; Kamitani and Tong, 2005)."

This conjecture was perhaps leading in 2005, but I don't think this is an accurate description of the state of affairs 13 years later, and it therefore seems a bit of a straw man. For example, a more nuanced view articulated by Swisher et al., 2010 from the Tong lab states that orientation information "can be found at spatial scales ranging from the size of individual columns to about a centimeter". I suggest the authors paint a more balanced picture in the introduction.

2) Subsection “Coarse-scale bias and stimulus vignetting: fMRI experiments”: "From this, we conclude that stimulus vignetting is a primary source of the coarse scale bias."

Why “a primary source”, rather than “a source”, or 'an important source'? It seems bold to conclude this based on a correlation of ~0.2-0.26 between the model and the data?

3) In the same section: "Decoding accuracy for the shifted between-modulator analysis was only slightly lower than within-modulator decoding accuracy".

Could the authors test whether this difference is statistically significant?

If the difference is significant, this could, as the authors point out, be due to the inner and outer radial edges. However, it could potentially also be caused by the fact that there is a small amount of orientation information present in the fMRI activity patterns that is not due to stimulus vignetting. The authors may also want to include this possibility in the text – even though they may find it unlikely.

4) Discussion section: "Our results provide a framework for reinterpreting a wide-range of findings on orientation selectivity, measured with both fMRI in human subjects and in single units."

This statement suggests that the study has wide implications for many studies that used decoding of oriented gratings. But it is left unspecified what those implications are. Could the authors be more specific here? For example, how should we reinterpret Kamitani and Tong, 2005 or Haynes and Rees, 2005? What wrong conclusions have been drawn, if we accept the notion that stimulus vignetting is the source of orientation decoding?

Reviewer #2:

The manuscript by Roth et al. uses computational modeling and experimental measurements to make a compelling case that stimulus edge effects (or 'vignetting') may be a substantial source of apparent orientation tuning as measured using standard-resolution fMRI. The work is technically solid and carefully presented, and the topic will be of significant interest to the fMRI community (given the widespread use of multivoxel pattern analysis). I have some comments, mainly conceptual/framing in nature, which should be relatively easy to address.

1) Wording and framing.

- There are a few places where I think the conclusions and claims should be toned down. For example, "vast number of previous studies" (Introduction section) and "wide-range of findings in the visual system". I assume the authors are referring to past studies that have used orientation stimuli in fMRI, and not the neurophysiology literature on orientation tuning in single neurons. While it is theoretically possible that vignetting effects may be influencing single-neuron response properties (since Gabor and grating patches are widely used as stimuli), it is not yet clear whether single-neuron studies need to be re-interpreted.

- The work of Carlson, 2014 involves modeling work that is similar to what is done in the current manuscript. Of course, the major advance of the present work is the demonstration of empirical findings, but this previous work might deserve more acknowledgment.

- Ultimately, a lot of the controversy regarding fMRI orientation decoding comes down to numbers, and it would be helpful to clarify what is meant by "coarse" and "fine". I assume the authors mean something to the effect that "coarse" is > 1 mm and "fine" is < 0.5 mm (or something like that).

2) Some big-picture perspective.

- The significance of the present work might be further emphasized by relating it more broadly to the general approach of MVPA (i.e. using linear classifiers to decode activity). I believe the larger lesson, highlighted by the present work, is that even seemingly simple properties are in fact hard to isolate experimentally and that powerful approaches like classification can pick up on aspects of the data that might not be what the experimenter intended. One way that we have conceptualized this (Naselaris, TICS, 2015) is that the orientation of a grating stimulus is not the only stimulus feature that can give rise to variance in data, and that classification can reflect a number of different stimulus features, such as those related to vignette effects, unless one does work to rule them out, e.g. by considering explicit computational models.

3) Acknowledgment of the limitations of the model.

- I think the main contribution of the current paper is the experimental results. The modeling analyses do provide value in that they demonstrate a concrete (and reasonably plausible) explanation of what could be driving the observed orientation-tuning results. However, as the authors recognize in the Discussion, there are many stimulus properties beyond what is characterized in the model that are known to affect V1 responses (e.g. surround suppression, contour integration, 2nd-order contrast effects, etc.), and which might also contribute to the orientation effects. Thus, the text could be clarified to acknowledge the limitations of the model and indicate what role the modeling results play in this specific paper. It seems that the role of the modeling results is to show concretely that imbalances in filter energy across orientations exist at stimulus edges and that this is one possible reason for finding orientation tuning in standard-resolution fMRI. (Note that I am not suggesting that the present paper needs to perform detailed model comparisons (in which different models are pitted to quantitatively account for individual voxel responses to a variety of stimulus conditions); that would be outside of the scope of this paper.)

4) Clarification of the modeled effect.

- It would be helpful to isolate and clarify the nature of the effect shown in Figure 2. One potential explanation is that for a linear filter stimulated with an optimal oriented grating, the filter shows a bigger response when the grating has a vignette edge orthogonal to the orientation compared to when the grating has a vignette edge parallel to the orientation. Is this the case in the model?

Reviewer #3:

Vignetting: interactions between grating and aperture edges might explain coarse-scale orientation-preference maps in V1 [I6 R8].

The orientation of a visual grating can be decoded from fMRI response patterns in primary visual cortex (Kamitani and Tong, 2005, Haynes and Rees, 2005). This was surprising, because fMRI voxels in these studies are 2-3 mm wide in each dimension and thus average over many columns of neurons responding to different orientations. Since then, many studies have sought to clarify why fMRI orientation decoding works so well.

The first explanation given was that even though much of the contrast of the neuronal orientation signals might cancel out in the averaging within each voxel, any given voxel might retain a slight bias toward certain orientations if it didn't sample all the columns exactly equally (e.g. Boynto,n 2005, Kamitani and Tong, 2005). By integrating the evidence across many slightly biased voxels with a linear decoder, it should then be possible to guess, better than chance, the orientation of the stimulus.

Another account (Op de Beeck, 2010, Freeman et al., 2011) proposed that decoding may rely exclusively on coarse-scale spatial patterns of activity. In particular, Freeman Brouwer, Heeger and Merriam, 2011 argued that orientations that are radial (aligned with a line that passes through the fixation point) are over-represented in the neural population. If this were the case, then a grating would elicit a coarse-scale response pattern across its representation in V1, in which the neurons representing edges pointing (approximately) at fixation are more strongly active. There is indeed evidence from multiple studies for a nonuniform representation of orientations in V1 (Furmanski and Engel, 2000, Sasaki et al., 2006, Serences et al., 2009, Mannion et al., 2010), perhaps reflecting the nonuniform probability distribution of orientation in natural visual experience. The over-representation of radial orientations might help explain the decodability of gratings. However, opposite-sense spirals (whose orientations are balanced about the radial orientation) are also decodable (Mannion et al., 2009, Alink et al., 2013). This might be due to a simultaneous over-representation of vertical orientations (Freeman et al., 2013, but see Alink et al., 2013).

There's evidence in favor of a contribution to orientation decoding of both coarse-scale (Freeman et al., 2011, Freeman et al., 2013) and fine-scale components of the fMRI patterns (e.g. Shmuel et al., 2010, Alink et al., 2013, Pratte et al., 2016, Alink et al., 2017).

Note that both coarse-scale and fine-scale pattern accounts suggest that voxels have biases in favor of certain orientations. An entirely novel line of argument altogether was introduced to the debate by Carlson, 2014.

Carlson, 2014 argued, on the basis of simulation results, that even if every voxel sampled a set of filters uniformly representing all orientations (i.e. without any bias), the resulting fMRI patterns could still reflect the orientation of a grating confined to a circular annulus (as standardly used in the literature). The reason lies in "the interaction between the stimulus region and the empty background" (Carlson, 2014), an effect of the relative orientations of the grating and the edge of the aperture (the annulus within which the grating is visible). Carlson's simulations showed that the average response of a uniform set of Gabor orientation filter is larger where the aperture edge is orthogonal to the grating. He also showed that the effect does not depend on whether the aperture edge is hard or soft (fading contrast). Because the voxels in this account have no bias in favour of particular orientations, Carlson aptly named his account an "unbiased" perspective.

The aperture edge adds edge energy. The effect is strongest when the edge is orthogonal to the carrier grating orientation. We can understand this in terms of the Fourier spectrum. Whereas a sine grating has a concentrated representation in the 2D Fourier amplitude spectrum, the energy is more spread out when an aperture limits the extent of the grating, with the effect depending on the relative orientations of grating and edge.

For an intuition on how this kind of thing can happen, consider a particularly simple scenario, where a coarse rectangular grating is limited by a sharp aperture whose edge is orthogonal to the grating. V1 cells with small receptive fields will respond to the edge itself as well as to the grating. When edge and grating are orthogonal, the widest range of orientation-selective V1 cells is driven. However, the effect is present also for sinusoidal gratings and soft apertures, where contrast fades gradually, e.g. according to a raised half-cosine.

An elegant new study by Roth, Heeger, and Merriam, 2018 now follows up on the idea of Carlson, 2014 with fMRI at 3T and 7T. Roth et al. refer to the interaction between the edge and the content of the aperture as "vignetting" and used apertures composed of either multiple annuli or multiple radial rays. These finer-grained apertures spread the vignetting effect all throughout the stimulated portion of the visual field and so are well suited to demonstrate the effect on decodability.

Roth et al. performed simulations, following Carlson, 2014 and assuming that every voxel uniformly samples all orientations. They confirm Carlson's account and show that the grating stimuli the group used earlier in Freeman et al., 2011 is expected to produce the stronger response to radial parts of the grating, where the aperture edge is orthogonal to the grating. Freeman et al., 2011 used a relatively narrow annulus (inner edge: 4.5°, outer edge: 9.5° eccentricity from fixation), where no part of the grating is far from the edge. This causes the vignetting effect to create the appearance of a radial bias that is strongest at the edges but present even in the central part of the annular aperture. Roth et al.'s present findings suggest that the group's earlier result might reflect vignetting, rather than (or in addition to) a radial bias of the V1 neurons.

Roth et al. use simulations also to show that their new stimuli, in which the aperture consists of multiple annuli or multiple radial rays, predict coarse-scale patterns across V1. They then demonstrate in single subjects measured with fMRI at 3T and 7T that V1 responds with the globally modulated patterns predicted by the account of Carlson, 2014.

The study is beautifully designed and expertly executed. Results compellingly demonstrate that, as proposed by Carlson, 2014, vignetting can account for the coarse-scale biases reported in Freeman et al., 2011. The paper also contains a careful discussion that places the phenomenon in a broader context. Vignetting describes a family of effects related to aperture edges and their interaction with the contents of the aperture. The interaction could be as simple as the aperture edge adding edge energy of a different orientation and thus changing orientation selective response. It could also involve extra-receptive-field effects such as non-isotropic surround suppression.

Another question is whether the vignetting effects Roth et al. demonstrate fully explain orientation decoding. The original study by Kamitani and Tong, 2005 used a wider annular aperture reaching further into the central region, where receptive fields are smaller (inner edge: 1.5° outer edge: 10° eccentricity from fixation). The interior parts of the stimulus may therefore not be affected by vignetting. Moreover, Wardle, Ritchie, Seymour, and Carlson, 2017 showed that vignetting is not necessary for orientation decoding.

Strengths

-Well-motivated and elegant stimulus design.

-3T and 7T fMRI data from a total of 14 subjects.

-Compelling results demonstrating that vignetting can cause coarse-scale patterns that enable orientation decoding,

Weaknesses

-The paper claims to introduce a novel idea that requires reinterpretation of a large literature. The claim of novelty is unjustified. Vignetting was discovered by Carlson et al., 2014 and in Wardle et al., 2017, Carlson's group showed that it may be one, but not the only contributing factor enabling orientation decoding. Carlson et al. deserve clearer credit throughout.

-The paper doesn't attempt to address whether decoding is still possible in the absence of vignetting effects, i.e. far from the aperture boundary.

Comments and suggestions

While the experiments and analyses are excellent and the paper well written, the current version is compromised by some exaggerated claims, suggesting greater novelty and consequence than is appropriate. This should be corrected.

Abstract: "Here, we show that a large body of research that purported to measure orientation tuning may have in fact been inadvertently measuring sensitivity to second-order changes in luminance, a phenomenon we term 'vignetting'."

Abstract: "Our results demonstrate that stimulus vignetting can wholly determine the orientation selectivity of responses in visual cortex measured at a macroscopic scale, and suggest a reinterpretation of a well-established literature on orientation processing in visual cortex."

Introduction: "Our results provide a framework for reinterpreting a wide-rangeof findings in the visual system."

Too strong of a claim of novelty. The effect beautifully termed "vignetting" here was discovered by Carlson, 2014, and that study deserves the credit for triggering a reevaluation of the literature, which began three years ago. The present study does place vignetting in a broader context, discussing a variety of mechanisms by which aperture edges might influence responses, but the basic idea, including that the key factor is the interaction between the edge and the grating orientation and that the edge need not be hard, are all introduced in Carlson, 2014. The present study very elegantly demonstrates the phenomenon with fMRI, but the effect has also previously been studied with fMRI by Wardle et al., 2017, so the fMRI component doesn't justify this claim, either. Finally, while results compellingly show that vignetting was a strong contributor in Freeman et al., 2011, they don't show that it is the only contributing factor for orientation decoding. In particular, Wardle et al., 2017 suggests that vignetting in fact is not necessary for orientation decoding.

Introduction: "We and others, using fMRI, discovered a coarse-scale orientation bias in human V1; each voxel exhibits an orientation preference that depends on the region of space that it represents (Furmanski and Engel, 2000; Sasaki et al., 2006; Mannion et al., 2010; Freeman et al., 2011; Freeman et al., 2013; Larsson et al., 2017). We observed a radial bias in the peripheral representation of V1: voxels that responded to peripheral locations near the vertical meridian tended to respond most strongly to vertical orientations; voxels along the peripheral horizontal meridian responded most strongly to horizontal orientations; likewise for oblique orientations. This phenomenon had gone mostly unnoticed previously. We discovered this striking phenomenon with fMRI because fMRI covers the entire retinotopic map in visual cortex, making it an ideal method for characterizing such coarse-scale representations."

A bit too much chest thumping. The radial-bias phenomenon was discovered by Sasaki et al., 2006. Moreover, the present study negates the interpretation in Freeman et al., 2011. Freeman et al., 2011 interpreted their results as indicating an over-representation of radial orientations in cortical neurons. According to the present study, the results were in fact an artifact of vignetting and whether neuronal biases played any role is questionable. Note that Freeman et al. used a narrower and more eccentric annulus than other studies (e.g. Kamitani and Tong, 2005), so may have been more susceptible to the vignetting artifact. The authors suggest that a large literature be reinterpreted, but apparently not their own study for which they specifically and compellingly show how vignetting probably affected it.

"A leading conjecture is that the orientation preferences in fMRI measurements arise primarily from random spatial irregularities in the fine-scale columnar architecture (Boynton, 2005; Haynes and Rees, 2005; Kamitani and Tong, 2005). […] On the other hand, we have argued that the coarse-scale orientation bias is the predominant orientation-selective signal measured with fMRI, and that multivariate decoding analysis methods are successful because of it (Freeman et al., 2011; Freeman et al., 2013). This conjecture [that coarse-scale orientation bias is the predominant signal] remains controversial because the notion that fMRI is sensitive to fine-scale neural activity is highly attractive, even though it has been proven difficult to validate empirically (Alink et al., 2013; Pratte et al., 2016; Alink et al., 2017)."

This passage is a bit biased. First, the present results question the interpretation of Freeman et al., 2011. While the authors' new interpretation (following Carlson, 2014) also suggests a coarse-scale contribution, it fundamentally changes the account. Moreover, the conjecture that coarse-scale effects play a role is not controversial. What is controversial is the claim that only coarse-scale effects contribute to fMRI orientation decoding. This extreme view is controversial not because it is attractive to think that fMRI can exploit fine-grained pattern information, but because the cited studies (Alink et al., 2013, Pratte et al., 2016, Alink et al., 2017, and additional studies, including Shmuel et al., 2010) present evidence in favor of a contribution from fine-grained patterns. The way these studies are cited would suggest to an uninformed reader that they provide evidence against a contribution from fine-grained patterns. More evenhanded language is in order here.

"the model we use is highly simplified; for example, it does not take intoaccount changes in spatial frequency tuning at greater eccentricities. Yet, despite the multiple sources of noise and the simplified assumptions of the model, the correspondence between the model's prediction and the empirical measurements are highly statistically significant. From this, we conclude that stimulus vignetting is a primary source of the course scale bias."

This argument is not compelling. A terrible model may explain a portion of the explainable variance that is minuscule, yet highly statistically significant. In the absence of inferential comparisons among multiple models and model checking (or a noise ceiling), better to avoid such claims.

Discussion: "One study (Alink et al., 2017) used inner and outer circular annuli, but added additional angular edges, the result of which should be a combination of radial and tangential biases. Indeed, this study reported that voxels had a mixed pattern of selectivity, with a considerable number of voxels reliably preferring tangential gratings, and other voxels reliably favoring radial orientations."

This reasoning makes sense. The additional edges between the patches (though perhaps not well described as vignetting) complicate the interpretation of the results of Alink et al., 2011. It would be good to check the strength of the effect by simulation. Happy to share the stimuli if someone wanted to look into this.

https://doi.org/10.7554/eLife.37241.012

Author response

Essential revisions:

I have provided a summary of essential revisions. You will find more details, as well as additional points that need to be addressed, in the original reviews (appended below).

1) Tone down claims of novelty:

-The paper claims to introduce a novel idea that requires reinterpretation of a large literature. The claim of novelty is unjustified. Vignetting was discovered by Carlson et al., 2014 and in Wardle et al., 2017, Carlson's group showed that it may be one, but not the only contributing factor enabling orientation decoding. Carlson et al. deserve clearer credit throughout.

See reviewer #2 point 1, and reviewer #3.

Carlson’s 2014 paper on stimulus edge effects was an insightful contribution to the field. His work inspired much of what we did here and we have edited our manuscript in several places to cite Carlson more prominently. We would like to point out how our theoretical framework differs from the work described in Carlson, 2014. Carlson used simulations to show that edge effects might contribute to orientation decoding, but he did not provide a mathematical/ theoretical explanation of those edge effects. This left open the possibility that Carlson’s simulation results might not have applied generally. It was even suggested that Carlson’s results were due to an artifact, a computing error, unrelated to the brain’s processing of orientation (Clifford and Mannion, 2015).

We have developed a mathematical/theoretical explanation for Carlson’s ‘edge effect’ based on the spread of Fourier power caused by the presence of a contrast change in the image. This is a much more general observation, which we call stimulus vignetting, and is actually much more than a simple edge artifact. We implemented an image-computable model of V1 that is capable of simulating responses to arbitrary stimuli, and we show that this model exhibits coarse-scale orientation-biases due to vignetting. The model has orientation-selective linear filters to simulate simple cells and energy filters (sum of squared responses of quadrature pairs of linear filters) for complex cells. The model was designed so that the filters evenly tile retinotopic space, orientation, and spatial frequency to avoid any possibility of the kind of artifact that was suggested to underlie Carlson’s simulation results. There are no so-called “2nd order filters” in our model. Yet it exhibits orientation-selective biases for 2nd-order modulation because of vignetting. This is an important theoretical result with widespread ramifications that go well beyond the interpretation of fMRI decoding (see below).

We tested the model’s predictions with empirical data and found a robust correspondence. Our empirical results completely contradict empirical results from Carlson’s lab (Wardle et al., 2017) in that we found measured fMRI responses could be entirely predicted by vignetting.

The effect of contrast changes on Fourier content has been known in the psychophysical literature for decades (Carter and Henning, 1971). Carlson pointed out that vignetting could give rise to the coarse-scale bias observed in Freeman et al., 2011. This possibility was, in fact, raised and tested by Freeman et al., 2011. Carlson’s results are fully consistent with our conclusion in Freeman et al., 2011 that coarse-scale orientation biases, and not a fine-scale bias from column sampling, are necessary and sufficient for orientation decoding.

2) Provide a more nuanced coverage of the literature. E.g., the conjecture that the orientation preferences in fMRI measurements arise primarily from random spatial irregularities in the fine-scale columnar architecture doesn't seem 'leading' anymore in 2018, and more nuanced positions have been articulated since.

See reviewer #1 point 1 and reviewer #3.

We have modified the text so as to describe the range of hypotheses that are currently in favor, ranging from those that believe that fMRI measurements at standard resolution are sensitive to fine-scale structure, to those who believe that fMRI at standard resolution is only sensitive to coarse-scale structure.

Our understanding of the literature is that the leading hypothesis is that orientation preferences in fMRI arise primarily from sampling of random fine-scale, columnar patterns of orientation selectivity. This hypothesis is assumed as the default explanation in several recent papers. For example, in a recent and highly cited (>200 citations in 6 years) review (Tong and Pratte, 2012), the authors state: “These orientation-decoding studies suggest that pattern analysis can be used to detect signals of columnar origin by pooling weakly feature-selective signals that can be found at the scale of millimeters, presumably due to variability in the organization of the columns.”. Reviewer 1 cites Swisher et al., 2010 as an example of a more nuanced view. The results presented in Swisher et al., 2010 do not provide evidence for the random-bias over the coarse-scale bias hypothesis. The reason is that the radial bias is broad-band, as is the polar angle component of the retinotopic map (Freeman et al., 2011). Yet, in a recent review, Tong and Pratte, 2012 write: “high-resolution functional imaging studies of the cat and human visual cortices have provided support for [the random bias] hypothesis (Swisher et al. 2010).” The random-bias hypothesis is still a leading conjecture today. We have replaced the above- mentioned sentence with the following one: “A number of previous studies have asserted that orientation preferences in fMRI measurements arise primarily from random spatial irregularities in the fine-scale columnar architecture (Boynton 2005, Haynes and Rees 2005, Kamitani and Tong, 2005)."

3) Discuss the broader implications of the study: the authors claim that the study has wide implications for many studies that used decoding of oriented gratings. But it is left unspecified what those implications are. Could the authors be more specific? For example, how should we reinterpret Kamitani and Tong, 2005 or Haynes and Rees, 2005? What wrong conclusions have been drawn, if we accept the notion that stimulus vignetting is the source of orientation decoding?

The significance of the present work might be further emphasized by relating it more broadly to the general approach of MVPA (i.e. using linear classifiers to decode activity). I believe the larger lesson, highlighted by the present work, is that even seemingly simple properties are in fact hard to isolate experimentally and that powerful approaches like classification can pick up on aspects of the data that might not be what the experimenter intended.

See reviewer #1 point 4, reviewer #2 point 2.

We have modified the Introduction and Discussion to address the broader implications of this study. We regret that the initial submission, unfortunately, was misleading.

The most important implication of our study is that stimulus vignetting could potentially impact any measurement of orientation selectivity, including single unit recording, calcium imaging, fMRI, and psychophysics. Most visual neuroscience experiments utilize stimuli that are restricted to a relatively small portion of the visual field. In such cases (i.e., nearly all vision experiments), vignetting can confound the interpretation of the results. The specific experimental method is irrelevant. If single-cell recordings are used to measure orientation tuning, but the receptive fields overlap the stimulus edge, or a 2nd-order modulator, then vignetting will alter the measured orientation selectivity. Vignetting, consequently, is relevant for the entire field of visual neuroscience. As we now explain in the Discussion, vignetting may underlie ostensible selectivity to second-order features such as the orientation of contrast changes (i.e. vignettes), illusory contours, and second-order texture patterns. We have now laid this out more clearly and explicitly in the Discussion section. We also suggest in the Discussion a possible reinterpretation of LGN orientation selectivity. Two recent studies, an fMRI study in humans and a calcium imaging study in mice, have claimed that orientation selectivity can be found in the LGN. However, vignetting was present in both studies. In the fMRI study the stimulus edge provided the vignette, and in the mice study the large receptive fields probably extended beyond the edge of the screen. The reinterpretation necessary, in these cases, involves considering the possibility that these studies were not necessarily measuring orientation sensitivity. Instead, the differential orientation responses could reflect solely vignetting. Therefore, it is possible that neurons in human and mouse LGN are not sensitive to orientation, as suggested originally by Hubel and Wiesel (1962).

As for fMRI orientation decoding studies, such as Kamitani and Tong, 2005 and Haynes and Rees, 2005, we have already shown that a coarse-scale bias is the dominant source of decodable orientation information (Freeman et al., 2011, 2013). In the current study we characterize vignetting as one of the sources of this coarse-scale bias. In the current study, we did not find evidence for any additional source for the coarse-scale bias, including a gain-map and/or asymmetric surround-suppression. But we cannot rule such sources out, as discussed in the Discussion section.

As now explained in the Discussion, we are no longer arguing about whether orientation decoding is dominated by coarse-scale orientation biases. We already know that it is. Note in particular the results reported by Freeman et al., 2013 in which we showed that orientation-decoding is unaffected when the slices are shifted by 1 mm (i.e., half the size of human V1 hypercolumns). We have no doubt that there are orientation columns in human V1. But we also have no doubt that orientation-decoding with fMRI at conventional resolutions is dominated by coarse-scale orientation bias, not the fine scale (columnar) architecture for orientation-selectivity. Otherwise, decoding would have suffered from shifting the slices by half the hypercolumn width. Rather than continuing to debate about whether fMRI decoding is dominated by coarse- or fine-scale information, the current study characterizes one of the sources of the coarse-scale bias, the implications of which are much broader than the technical details of orientation-decoding with fMRI (see above).

Leaving aside this debate about whether orientation decoding with fMRI depends on coarse- vs. fine-scale information, it is important to remember that the contribution of many decoding studies remains, irrespective of the source of decodable information. For example, a primary contribution of Kamitani and Tong, 2005 regards feature-based attention: they showed that attending to one or the other orientation changes the activity in V1 in such a way that the attended orientation can be decoded. That result holds regardless of whether the decoding depends on coarse- vs fine-scale. Similarly, Haynes and Rees, 2005 showed that V1 activity reflects properties of invisible stimuli, using decoding. Likewise for many studies that used decoding, forward modeling, and other MVPA techniques to reveal properties of cortical processing. Most, if not all, of these findings do not depend on the source of decodable information.

4) Address/discuss whether decoding is still possible in the absence of vignetting effects, or is solely dependent on vignetting. See reviewer #1 point 2, 3; reviewer #3.

The experiment was not designed to address this question, and we cannot definitively answer this question. It is possible — we think likely — that vignetting is the sole source of information for decoding orientation from fMRI measurements at conventional resolutions. But it is also possible that there are additional sources to the coarse-scale bias, such as a gain-map and/or asymmetric surround suppression, as discussed in the Discussion section. Our previous studies on coarse-scale biases (Freeman et al., 2011, and Freeman et al., 2013) provide more than enough evidence that orientation decoding is dominated by coarse-scale orientation biases.

Other investigators continue to disagree with this conclusion, but we have yet to see convincing evidence that fine-scale features contribute to orientation decoding at conventional fMRI resolutions, and the goal of our study was not to look for such evidence.

Can a simple experiment, consisting of full-field gratings answer this question? It may seem so, since large stimuli should evoke vignetting only at the far periphery, so that any orientation bias at mid-eccentricities would appear to be the result of neural orientation selectivity. Specifically, reviewers 1 and 3 suggest using stimuli covering a wider portion of the visual field, which should push the coarse-scale bias associated with vignetting out to the far periphery. However, such an experiment cannot, in fact, determine whether or not orientation information is available in the absence of vignetting. The reason is that, in such an experiment, one cannot rule out the possibility that voxels with population receptive fields centered at mid-eccentricities respond to the stimulus and/or screen edge. In other words, pRFs modeled as simple Gaussian functions, often have heavy tails so that they respond, albeit with small amplitude, to stimuli presented far away from their centers. Reviewer 3, in his online version of the review, suggested dividing V1 into voxels that do respond to the stimulus edge and voxels that do not. If vignetting is the source of orientation information, then an ROI of voxels that do respond to the edge should enable orientation decoding, while an ROI of voxels that do not respond to the edge should not support decoding. But the logic of the approach suggested by reviewer 3 is statistically flawed. In a standard fMRI localizer we search for voxels that respond to a stimulus, and therefore our null hypothesis is that they do not respond. One generally uses a high threshold, after multiple- comparison correction, to minimize type 1, or false positive, errors. Attempting to select voxels that do not respond to the stimulus in such an experiment amounts to accepting the null hypothesis, i.e., an elementary statistical error. Since decoding methods are tailored to exploit weak signals, sub-threshold voxels exhibiting vignetting may support decoding.

More importantly, this debate about whether decoding is still possible in the absence of vignetting entirely misses the point. Rather than continuing to debate about whether fMRI decoding is possible without coarse-scale information, the current study characterizes one of the sources of the coarse-scale bias, the implications of which are much broader than the technical details of orientation-decoding with fMRI (see above).

Reviewer #1:

[…] 1) Introduction section: "A leading conjecture is that the orientation preferences in fMRI measurements arise primarily from random spatial irregularities in the fine-scale columnar architecture (Boynton, 2005; Haynes and Rees, 2005; Kamitani and Tong, 2005)."

This conjecture was perhaps leading in 2005, but I don't think this is an accurate description of the state of affairs 13 years later, and it therefore seems a bit of a straw man. For example, a more nuanced view articulated by Swisher et al., 2010 from the Tong lab states that orientation information "can be found at spatial scales ranging from the size of individual columns to about a centimeter". I suggest the authors paint a more balanced picture in the introduction.

See Essential revisions #2.

2) Subsection “Coarse-scale bias and stimulus vignetting: fMRI experiments”: "From this, we conclude that stimulus vignetting is a primary source of the coarse scale bias."

Why “a primary source”, rather than “a source”, or 'an important source'? It seems bold to conclude this based on a correlation of ~0.2-0.26 between the model and the data?

We have removed this sentence from the text.

3) In the same section: "Decoding accuracy for the shifted between-modulator analysis was only slightly lower than within-modulator decoding accuracy".

Could the authors test whether this difference is statistically significant?

If the difference is significant, this could, as the authors point out, be due to the inner and outer radial edges. However, it could potentially also be caused by the fact that there is a small amount of orientation information present in the fMRI activity patterns that is not due to stimulus vignetting. The authors may also want to include this possibility in the text – even though they may find it unlikely.

We now include this possibility in the text: “However, we cannot rule out the possibility that some other orientation information, that is not dependent on the vignette, is present in the data.” (The difference is significant, according to the phase permutation test.)

4) Discussion section: "Our results provide a framework for reinterpreting a wide-range of findings on orientation selectivity, measured with both fMRI in human subjects and in single units."

This statement suggests that the study has wide implications for many studies that used decoding of oriented gratings. But it is left unspecified what those implications are. Could the authors be more specific here? For example, how should we reinterpret Kamitani and Tong, 2005 or Haynes and Rees, 2005? What wrong conclusions have been drawn, if we accept the notion that stimulus vignetting is the source of orientation decoding?

See Essential revisions #3.

Reviewer #2:

[…] 1) Wording and framing.

- There are a few places where I think the conclusions and claims should be toned down. For example, "vast number of previous studies" (Introduction section) and "wide-range of findings in the visual system". I assume the authors are referring to past studies that have used orientation stimuli in fMRI, and not the neurophysiology literature on orientation tuning in single neurons. While it is theoretically possible that vignetting effects may be influencing single-neuron response properties (since Gabor and grating patches are widely used as stimuli), it is not yet clear whether single-neuron studies need to be re-interpreted.

See Essential revisions #3. Actually, we are referring to the literature on orientation tuning in single neurons, in addition to other methods, including fMRI. We agree that further empirical demonstrations of vignetting, using various measurement methods, would be helpful. However, based on our theoretical, computational, and fMRI results, we believe it is quite clear that vignetting may have affected many measurements of orientation selectivity, at various spatial scales including single-unit recordings.

- The work of Carlson, 2014 involves modeling work that is similar to what is done in the current manuscript. Of course, the major advance of the present work is the demonstration of empirical findings, but this previous work might deserve more acknowledgment.

We agreed with the reviewer’s comment regarding our empirical findings. See Essential revisions #1 for clarification about our views regarding the novelty of the mathematical/ theoretical framework that we describe in the manuscript.

- Ultimately, a lot of the controversy regarding fMRI orientation decoding comes down to numbers, and it would be helpful to clarify what is meant by "coarse" and "fine". I assume the authors mean something to the effect that "coarse" is > 1 mm and "fine" is < 0.5 mm (or something like that).

The precise distinction between coarse and fine orientation biases has no bearing on our results or conclusions. Surely, any distinction we choose would be challenged by one researcher or another. See Essential revisions #3.

2) Some big-picture perspective.

- The significance of the present work might be further emphasized by relating it more broadly to the general approach of MVPA (i.e. using linear classifiers to decode activity). I believe the larger lesson, highlighted by the present work, is that even seemingly simple properties are in fact hard to isolate experimentally and that powerful approaches like classification can pick up on aspects of the data that might not be what the experimenter intended. One way that we have conceptualized this (Naselaris TICS, 2015) is that the orientation of a grating stimulus is not the only stimulus feature that can give rise to variance in data, and that classification can reflect a number of different stimulus features, such as those related to vignette effects, unless one does work to rule them out, e.g. by considering explicit computational models.

3) Acknowledgment of the limitations of the model.

- I think the main contribution of the current paper is the experimental results. The modeling analyses do provide value in that they demonstrate a concrete (and reasonably plausible) explanation of what could be driving the observed orientation-tuning results. However, as the authors recognize in the Discussion, there are many stimulus properties beyond what is characterized in the model that are known to affect V1 responses (e.g. surround suppression, contour integration, 2nd-order contrast effects, etc.), and which might also contribute to the orientation effects. Thus, the text could be clarified to acknowledge the limitations of the model and indicate what role the modeling results play in this specific paper. It seems that the role of the modeling results is to show concretely that imbalances in filter energy across orientations exist at stimulus edges and that this is one possible reason for finding orientation tuning in standard-resolution fMRI. (Note that I am not suggesting that the present paper needs to perform detailed model comparisons (in which different models are pitted to quantitatively account for individual voxel responses to a variety of stimulus conditions); that would be outside of the scope of this paper.)

We added a paragraph to the Discussion to clarify the purpose of the model. The purpose of the model was to: 1) demonstrate that stimulus vignetting is sufficient to create a radial bias, 2) create novel stimuli with predicted opposite biases, and 3) provide predictions of single voxel responses in response to the novel stimuli; the model is image-computable and accepts arbitrary stimuli as input. The strength of the model is its simplicity. Our goal is not to provide an accurate and complete model of V1 activity, but rather to provide a platform for assessing vignetting. We purposely kept it simple so that we could be certain that vignetting is indeed the source of orientation bias in the model. The point is that ostensibly interesting features of V1 responses might actually be a result of vignetting. For example, sensitivity of V1 neurons to second-order changes might actually reflect vignetting. If we were to include asymmetric surround suppression in the model, any orientation bias could be a combined result of vignetting and surround suppression.

4) Clarification of the modeled effect

- It would be helpful to isolate and clarify the nature of the effect shown in Figure 2. One potential explanation is that for a linear filter stimulated with an optimal oriented grating, the filter shows a bigger response when the grating has a vignette edge orthogonal to the orientation compared to when the grating has a vignette edge parallel to the orientation. Is this the case in the model?

A linear filter will not necessarily respond more strongly to an orthogonal vignette vs. a parallel vignette. That would depend on the specific neuron’s receptive field. For example, in Figure 1, the neuron with a receptive field depicted in green would respond more to the vertical edge than to the horizontal edge, while the red neuron would respond more strongly to the horizontal edge.

Reviewer #3:

[…] The study is beautifully designed and expertly executed. Results compellingly demonstrate that, as proposed by Carlson, 2014, vignetting can account for the coarse-scale biases reported in Freeman et al., 2011. The paper also contains a careful discussion that places the phenomenon in a broader context. Vignetting describes a family of effects related to aperture edges and their interaction with the contents of the aperture. The interaction could be as simple as the aperture edge adding edge energy of a different orientation and thus changing orientation selective response. It could also involve extra-receptive-field effects such as non-isotropic surround suppression.

Another question is whether the vignetting effects Roth et al. demonstrate fully explain orientation decoding. The original study by Kamitani and Tong, 2005 used a wider annular aperture reaching further into the central region, where receptive fields are smaller (inner edge: 1.5° outer edge: 10° eccentricity from fixation). The interior parts of the stimulus may therefore not be affected by vignetting. Moreover, Wardle, Ritchie, Seymour, and Carlson, 2017 showed that vignetting is not necessary for orientation decoding.

See Essential revisions #4

Strengths

-Well-motivated and elegant stimulus design.

-3T and 7T fMRI data from a total of 14 subjects.

-Compelling results demonstrating that vignetting can cause coarse-scale patterns that enable orientation decoding.

Weaknesses

-The paper claims to introduce a novel idea that requires reinterpretation of a large literature. The claim of novelty is unjustified. Vignetting was discovered by Carlson et al., 2014 and in Wardle et al., 2017, Carlson's group showed that it may be one, but not the only contributing factor enabling orientation decoding. Carlson et al. deserve clearer credit throughout.

See Essential revisions #1

-The paper doesn't attempt to address whether decoding is still possible in the absence of vignetting effects, i.e. far from the aperture boundary.

See Essential revisions #4

Comments and suggestions

While the experiments and analyses are excellent and the paper well written, the current version is compromised by some exaggerated claims, suggesting greater novelty and consequence than is appropriate. This should be corrected.

Abstract: "Here, we show that a large body of research that purported to measure orientation tuning may have in fact been inadvertently measuring sensitivity to second-order changes in luminance, a phenomenon we term 'vignetting'."

Abstract: "Our results demonstrate that stimulus vignetting can wholly determine the orientation selectivity of responses in visual cortex measured at a macroscopic scale, and suggest a reinterpretation of a well-established literature on orientation processing in visual cortex."

Introduction: "Our results provide a framework for reinterpreting a wide-rangeof findings in the visual system."

We stand by these statements.

Too strong of a claim of novelty. The effect beautifully termed "vignetting" here was discovered by Carlson, 2014, and that study deserves the credit for triggering a reevaluation of the literature, which began three years ago. The present study does place vignetting in a broader context, discussing a variety of mechanisms by which aperture edges might influence responses, but the basic idea, including that the key factor is the interaction between the edge and the grating orientation and that the edge need not be hard, are all introduced in Carlson, 2014. The present study very elegantly demonstrates the phenomenon with fMRI, but the effect has also previously been studied with fMRI by Wardle et al., 2017, so the fMRI component doesn't justify this claim, either. Finally, while results compellingly show that vignetting was a strong contributor in Freeman et al., 2011, they don't show that it is the only contributing factor for orientation decoding. In particular, Wardle et al., 2017 suggests that vignetting in fact is not necessary for orientation decoding.

See Essential revision #1.

Indeed, our results contradict those of Wardle et al., 2017.

We are not sure which “claim of novelty” the reviewer is referring to. All excerpts seem accurate to us.

Introduction: "We and others, using fMRI, discovered a coarse-scale orientation bias in human V1; each voxel exhibits an orientation preference that depends on the region of space that it represents (Furmanski and Engel, 2000; Sasaki et al., 2006; Mannion et al., 2010; Freeman et al., 2011; Freeman et al., 2013; Larsson et al., 2017).[…] This phenomenon had gone mostly unnoticed previously. We discovered this striking phenomenon with fMRI because fMRI covers the entire retinotopic map in visual cortex, making it an ideal method for characterizing such coarse-scale representations."

A bit too much chest thumping. The radial-bias phenomenon was discovered by Sasaki et al., 2006. Moreover, the present study negates the interpretation in Freeman et al., 2011. Freeman et al., 2011 interpreted their results as indicating an over-representation of radial orientations in cortical neurons. According to the present study, the results were in fact an artifact of vignetting and whether neuronal biases played any role is questionable. Note that Freeman et al. used a narrower and more eccentric annulus than other studies (e.g. Kamitani and Tong, 2005), so may have been more susceptible to the vignetting artifact. The authors suggest that a large literature be reinterpreted, but apparently not their own study for which they specifically and compellingly show how vignetting probably affected it.

We cite Sasaki, 2006 along with an earlier paper by Furmanski and Engel, 2000, in the opening sentence of the second paragraph of the Introduction, along with our own papers. We do not prioritize our own work over these other papers.

We completely agree that our previous papers, including Freeman et al., 2011 were susceptible to vignetting. In fact, our claim in the current paper is that the results reported in our previous papers were dominated by vignetting. It is true that Freeman et al. suggested that the coarse- scale bias may reflect a gain-map. However, this suggestion was presented as just that – a suggestion that was made in the Discussion section of the 2011 paper. Indeed, in our current study we found no evidence for a gain-map, but as we state in the Discussion section of the current manuscript, we cannot (based on the current results) rule out additional sources of the coarse-scale bias, including a gain-map. In fact, the reviewer seems to be aware of this (“while results compellingly show that vignetting was a strong contributor in Freeman et al., 2011, they don't show that it is the only contributing factor”, two paragraphs above).

"A leading conjecture is that the orientation preferences in fMRI measurements arise primarily from random spatial irregularities in the fine-scale columnar architecture (Boynton, 2005; Haynes and Rees, 2005; Kamitani and Tong, 2005). […] On the other hand, we have argued that the coarse-scale orientation bias is the predominant orientation-selective signal measured with fMRI, and that multivariate decoding analysis methods are successful because of it (Freeman et al., 2011; Freeman et al., 2013). This conjecture [that coarse-scale orientation bias is the predominant signal] remains controversial because the notion that fMRI is sensitive to fine-scale neural activity is highly attractive, even though it has been proven difficult to validate empirically (Alink et al., 2013; Pratte et al., 2016; Alink et al., 2017)."

This passage is a bit biased. First, the present results question the interpretation of Freeman et al., 2011. While the authors' new interpretation (following Carlson, 2014) also suggests a coarse-scale contribution, it fundamentally changes the account. Moreover, the conjecture that coarse-scale effects play a role is not controversial. What is controversial is the claim that only coarse-scale effects contribute to fMRI orientation decoding. This extreme view is controversial not because it is attractive to think that fMRI can exploit fine-grained pattern information, but because the cited studies (Alink et al., 2013, Pratte et al., 2016, Alink et al., 2017, and additional studies, including Shmuel et al., 2010) present evidence in favor of a contribution from fine-grained patterns. The way these studies are cited would suggest to an uninformed reader that they provide evidence against a contribution from fine-grained patterns. More evenhanded language is in order here.

We have changed the text to read: “However, the finding that a coarse-scale bias is the source of orientation decoding remains controversial, and several recent studies have attempted to disprove it (Alink, Krugliak et al., 2013, Pratte, Sy et al., 2016, Alink, Walther et al., 2017), in part, we believe, because the notion that fMRI is sensitive to fine-scale neural activity is highly attractive.”

"the model we use is highly simplified; for example, it does not take intoaccount changes in spatial frequency tuning at greater eccentricities. Yet, despite the multiple sources of noise and the simplified assumptions of the model, the correspondence between the model's prediction and the empirical measurements are highly statistically significant. From this, we conclude that stimulus vignetting is a primary source of the course scale bias."

This argument is not compelling. A terrible model may explain a portion of the explainable variance that is minuscule, yet highly statistically significant. In the absence of inferential comparisons among multiple models and model checking (or a noise ceiling), better to avoid such claims.

We have removed this sentence from the text.

Discussion: "One study (Alink et al., 2017) used inner and outer circular annuli, but added additional angular edges, the result of which should be a combination of radial and tangential biases. Indeed, this study reported that voxels had a mixed pattern of selectivity, with a considerable number of voxels reliably preferring tangential gratings, and other voxels reliably favoring radial orientations."

This reasoning makes sense. The additional edges between the patches (though perhaps not well described as vignetting) complicate the interpretation of the results of Alink et al., 2011. It would be good to check the strength of the effect by simulation. Happy to share the stimuli if someone wanted to look into this.

We intend to share publicly the code for our model so that the reviewer (or anyone else) can follow up.

https://doi.org/10.7554/eLife.37241.013

Article and author information

Author details

  1. Zvi N Roth

    Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, United States
    Contribution
    Data curation, Formal analysis, Investigation, Visualization, Methodology, Writing–original draft, Writing–review and editing, Software, Validation
    For correspondence
    zvi.roth@mail.huji.ac.il
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-2173-1625
  2. David J Heeger

    1. Department of Psychology, New York University, New York, United States
    2. Center for Neural Science, New York University, New York, United States
    Contribution
    Conceptualization, Software, Investigation, Writing—original draft, Writing—review and editing, Funding acquisition, Formal analysis
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0002-3282-9898
  3. Elisha P Merriam

    Laboratory of Brain and Cognition, National Institute of Mental Health, National Institutes of Health, Bethesda, United States
    Contribution
    Conceptualization, Methodology, Software, Investigation, Resources, Writing—original draft, Writing—review and editing, Supervision, Project administration, Funding acquisition, Formal analysis, Data curation, Validation
    Competing interests
    No competing interests declared
    ORCID icon "This ORCID iD identifies the author of this article:" 0000-0003-2787-566X

Funding

National Institute of Mental Health (ZIA-MH-002909)

  • Zvi N. Roth
  • Elisha P. Merriam

Hebrew University of Jerusalem (ELSC Postdoctoral Fellowship Abroad)

  • Zvi N. Roth

National Institutes of Health (R01-EY025673)

  • David J. Heeger
  • Elisha P. Merriam

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

Supported by NIH grants R01-EY025673, and by the Intramural Research Program of the National Institutes of Health (ZIA-MH-002909) - National Institute of Mental Health Clinical Study Protocol 93-M-0170, NCT00001360. ZNR was supported by The ELSC Postdoctoral Fellowship Abroad, Hebrew University of Jerusalem. Special thanks to Eero Simoncelli, Chris Baker, Peter Bandettini for helpful comments.

Ethics

Human subjects: Experiments were conducted with the written consent of each subject and in accordance with the safety guidelines for fMRI research, as approved by the University Committee on Activities Involving Human Subjects at New York University and by the Institutional Review Board of the National Institutes of Health.

Senior Editor

  1. Sabine Kastner, Princeton University, United States

Reviewing Editor

  1. Floris P de Lange, Donders Institute for Brain, Cognition and Behaviour, Netherlands

Publication history

  1. Received: April 3, 2018
  2. Accepted: July 1, 2018
  3. Version of Record published: August 14, 2018 (version 1)

Copyright

This is an open-access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.

Metrics

  • 1,194
    Page views
  • 134
    Downloads
  • 1
    Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.

Download links

A two-part list of links to download the article, or parts of the article, in various formats.

Downloads (link to download the article as PDF)

Download citations (links to download the citations from this article in formats compatible with various reference manager tools)

Open citations (links to open the citations from this article in various online reference manager services)

  1. Further reading

Further reading

    1. Neuroscience
    Rose A Cooper, Maureen Ritchey
    Research Article
    1. Neuroscience
    Austin D Hocker et al.
    Research Article