Deploying goal-directed spatial attention towards visual locations allows observers to detect targets with higher accuracy (Hawkins et al., 1990), faster reaction times (Posner, 1980), and higher sensitivity (Sagi and Julesz, 1986) providing humans and non-human primates with a mechanism to select and prioritize spatial visual information (Carrasco, 2011). These enhanced behavioral responses are accompanied by both an increase in the gain of sensory responses near attended locations (Connor et al., 1996; McAdams and Maunsell, 1999) and changes in the shape and size of receptive fields, typically shrinking and shifting towards the target of attention (Ben Hamed et al., 2002; Womelsdorf et al., 2006; Anton-Erxleben et al., 2009; Klein et al., 2014; Kay et al., 2015; Vo et al., 2017; van Es et al., 2018). These changes in neural representation are thought to contribute to behavioral enhancement, but because both gain and changes in spatial properties co-occur in biological systems, it is not possible to disentangle them. Computational models of the visual system allow us to design experiments to independently examine the effects of such changes (Lindsay and Miller, 2018; Eckstein et al., 2000).

Shrinkage and shift of receptive fields toward attended targets has been observed in both single unit (Womelsdorf et al., 2006; Anton-Erxleben et al., 2009) and population (Klein et al., 2014; Vo et al., 2017; Fischer and Whitney, 2009; van Es et al., 2018) activity, and has been suggested to lead to behavioral enhancement through a variety of possible mechanisms (Anton-Erxleben and Carrasco, 2013). For example, receptive field changes might magnify the cortical representation of attended regions (Moran and Desimone, 1985), select for relevant information (Anton-Erxleben et al., 2009; Sprague and Serences, 2013), reduce uncertainty about spatial position (Vo et al., 2017), increase spatial discriminability (Kay et al., 2015; Fischer and Whitney, 2009), or change estimates of perceptual size (Anton-Erxleben et al., 2007). Compression of visual space is also observed just prior to saccades and thought to shift receptive fields towards the saccade location (Zirnsak et al., 2014; Colby and Goldberg, 1999; Merriam et al., 2007) and maintain a stable representation of visual space (Kusunoki and Goldberg, 2003; Tolias et al., 2001; Ross et al., 1997; Duhamel et al., 1992).

Shrinkage and shift of receptive fields has also been hypothesized to occur as a side effect of increasing gain of neural responses (Klein et al., 2014; Compte and Wang, 2006), thus raising the question of which of these physiological effects could be responsible for enhanced perception. When gain is asymmetric across a receptive field, the overall effect will be to shift the receptive field location towards the side with the largest gain. Similarly, asymmetric gain can be expected to change spatial tuning properties such as the size and structure of the receptive field. These concomitant changes of receptive field size, location, and structure could improve perceptual performance through the mechanisms described above, or could be an epiphenomenological consequence of increasing gain. Increasing gain by itself has also been hypothesized to be a mechanism for improving perceptual performance, because response gain can increase the signal-to-noise ratio and make responses to different stimuli more discriminable (McAdams and Maunsell, 1999; Cohen and Newsome, 2008). Moreover, larger responses for attended stimuli can act to select relevant information when read-out through winner-take-all mechanisms (Lee et al., 1999; Pelli, 1985; Pestilli et al., 2011; Palmer et al., 2000; Hara et al., 2014).

We took a modeling approach to ask what effects gain changes incur on spatial receptive field structure when introduced at the earliest stage of visual processing and to ask which effects would improve behavioral performance. We modified a convolutional neural network (CNN) trained on ImageNet categorization to test various hypotheses by implementing them as elements of the model architecture. CNN architectures can be designed to closely mimic the primate visual hierarchy (Yamins et al., 2014; Kubilius et al., 2018). Training ‘units’ in these networks to categorize images leads to visual filters that show a striking qualitative resemblance to the filters observed in early visual cortex (Krizhevsky et al., 2012) and the pattern of activity of these units when presented with natural images is sufficient to capture a large portion of the variance in neural activity in the retina (McIntosh et al., 2016), in early visual cortex (Cadena et al., 2019), and in later areas (Güçlü and van Gerven, 2015; Cichy et al., 2016; Eickenberg et al., 2017; Khaligh-Razavi and Kriegeskorte, 2014; Yamins et al., 2014). Cortical responses and neural network activity also share a correlation structure across natural image categories (Storrs et al., 2020). These properties of CNNs make them a useful tool which we can use to indirectly study visual cortex, probing activity and behavior in ways that are impractical in humans and non-human primates (Lindsay and Miller, 2018).

Using simulations based on a CNN observer model we found that gain changes introduced at the earliest stage in visual processing improved task performance with a magnitude comparable to that measured in human subjects. While these gain changes also induced changes in receptive field location, size, and spatial structure similar to that reported in physiological measurements, these changes were neither necessary nor sufficient for improving model task performance. More specifically, we designed a simple cued object-detection task and measured improved human performance on trials with focal attention. Using CORnet-Z (Kubilius et al., 2018), a CNN whose architecture was designed to maximize similarity with the primate visual stream, we measured a similar improvement in detection performance when a Gaussian gain augmented inputs coming from a ‘cued’ location. We found that the network mirrored the physiology of human and non-human primates: units shifted their center-of-mass toward the locus of attention and shrank in size, all in a gain-dependent manner. We isolated each of these physiological changes to determine which, if any, could account for the benefits to performance. A model with only gain reproduced the benefits of cued attention while models with only receptive field shifts, shrinkage, or only changes in receptive field structure were unable to provide any benefit to task performance. These results held for both an object detection task and a category discrimination task. Gain applied or removed at the last stage of processing in the CNN observer model demonstrated that gain was both necessary and sufficient to account for the benefits in task performance of the model.

We characterized the ability of human observers to detect objects in a grid of four images, with or without prior information about the object’s possible location (Figure 1). Observers were given a written category label, for example ‘ferris wheel’, and shown five exemplar images of that category (Category intro, Figure 1a). This was followed by a block of 80 trials in which observers tried to detect the presence or absence of the target category among the four images in the grid (Each trial, Figure 1a). Half of the 80 trials had focal cues and 50% of the focal (and distributed) trials included a target image. On focal trials, a cue indicated with 100% validity the grid quadrant that could contain a target while on distributed trials no information was given as to where an image of the target category could appear. Distractor images were randomly sampled from the nineteen non-target image categories. Stimulus durations were sampled uniformly from 1 (8.3 ms), 2 (16.7), 4 (33.3), 8 (66.7), 16 (133.3), or 32 (267.7) frames (Stimulus, 8.3ms per frame, Figure 1a). Image grids were masked before and after stimulus presentation by shuffling the pixel locations in the stimulus images, ensuring that the luminance during each trial remained constant. Observers had 2 s to make a response and each trial was followed by a 0.25 s inter-trial interval. Observers completed one training block on an unused category prior to data collection.

Figure 1

Download asset Open asset

Human observers improved their performance on this detection task when given a focal cue indicating the potential location of a target (Figure 1b). We quantified human performance by computing sensitivity, d’, as a function of stimulus duration separately for focal and distributed conditions. Across all observers the $d^{\prime }$ function was best fit as:

where $\alpha$ scaled the function for the focal condition. At a stimulus duration of 8.3ms (one frame) observers were near chance performance regardless of cueing condition. On distributed trials observers exceeded threshold performance ($d^{\prime }=1$) at a stimulus duration of 155ms, 95% CI [135, 197]. For focal trials, the same threshold was reached with only a 38ms [32, 43] stimulus duration, demonstrating a substantial performance benefit of the focal cue. We found that $d^{\prime }$ in the focal condition was higher than in the distributed condition, average increase across observers $\alpha =1.67\times$ [1.57, 1.74].

Using a drift diffusion model, we found that the majority of this performance benefit came from improved perceptual sensitivity, rather than speed-accuracy trade off. We assessed this by fitting a drift diffusion model to the reaction time and choice data (Wagenmakers et al., 2007). Drift diffusion models assume that responses are generated by a diffusion process in which evidence accumulates over time toward a bound. We used the equations in Wagenmakers et al., 2007 to transform each observer’s percent correct, mean reaction time, and reaction time variance for the twenty categories and two focal conditions into drift rate, bound separation, and non-decision time. The drift rate parameter is designed to isolate the effect of external input, the non-decision time reflects an internal delay before stimulus processing begins, and the bound separation is a proxy for how conservative observers are. Comparing the drift rate parameter we observed a similar effect to what was described above for $d^{\prime }$: the average drift rate across observers in the focal condition was 1.61×, 95% CI [1.39, 1.77] the drift rate in the distributed condition. This suggests that the majority of the performance gain observed in the $d^{\prime }$ parameter came from increased stimulus information. We did find that the other parameters of the drift diffusion model were also sensitive to duration and condition, but in opposite directions. We found larger bound separation at longer stimulus durations and on focal trials (focal bound-separation 1.57× distributed [1.37, 1.75]), consistent with observers being more conservative on trials where more information was available. But this increase in cautiousness was offset by a shorter non-decision time on focal trials (0.26 s) compared to distributed (0.38, [0.34, 0.41]).

Having shown that a spatial cue provides human observers with increased stimulus information in this task, we next sought to show that a neural network model of the human visual stream could replicate this behavior under similar conditions. We used a convolutional neural network (CNN) model, CORnet-Z (Kubilius et al., 2018), a neural network designed to mimic primate V1, V2, V4, and IT and optimized to perform object recognition for images at a similar scale to our task. CORnet-Z is a four-layer CNN with repeated convolutional, rectified linear units (ReLU), and pooling (Figure 2d). We used pretrained weights which were optimized for object categorization on ImageNet (Deng et al., 2009). To perform our category detection task, we added a fully connected output layer for each category and trained the weights of that layer to predict the presence of the twenty object categories selected for this study, thus creating a neural network observer model, that is a model designed to idealize the computations human observers perform in the four-quadrant object detection task. We applied the observer model to a task analogous to the one human observers performed (Figure 2c). The prediction layers added to the end of the model provided independent readouts for the presence or absence of the different target categories (Linear classifier, Figure 2c). These output layers were trained on a held out set of full-size images from each category. On a separate held out validation set of 100 images, the trained prediction layers achieved a median AUC of 0.90, range [0.77, 0.96].

Figure 2

Download asset Open asset

To examine the computational mechanisms that could underlie the performance benefit of focal cues we added a multiplicative Gaussian gain centered at the location of the cued image (Figure 2b, Gaussian width 56 px). We applied this gain at the first layer of the model and tested various strengths of gain.

To align the human and model performance we took the performance of the model in the distributed condition (Distributed, Figure 2a) and found the stimulus duration at which observers in the distributed condition of the human data matched this performance level (64ms, Figure 1b). We then scaled up the amplitude of the Gaussian gain incrementally and found that we could mimic the performance enhancement of human spatial attention by setting the maximum of the Gaussian gain field to approximately $4\times$. The model with this level of gain had a median AUC across categories of 0.80, 95% CI [0.77, 0.82] compared to 0.71 [0.67, 0.72] without gain and a median AUC improvement of 0.09 [0.08, 0.12] within each category.

The gain strengths necessary to induce an increase in task performance in the neural network observer model were relatively large compared to the gain due to directed attention observed in measurements of single unit (Luck et al., 1997; Treue and Martínez Trujillo, 1999) and population (Birman and Gardner, 2019) activity. We attribute this difference to the lack of any non-linear “winner-take-all” type of activation in the CNN. In the primate visual system, it is thought that non-linearities such as exponentiation and normalization can accentuate response differences (Reynolds and Heeger, 2009; Carandini and Heeger, 2012) and act as a selection mechanism for sensory signals (Pestilli et al., 2011). We tested whether similar non-linear mechanisms would allow for smaller gain strengths to be amplified to the range needed by our model. This was tested by raising the activations of units to an exponent before re-normalizing the activation of all units at the output of each layer (see Materials and methods for details). This has the effect of amplifying active units and further suppressing inactive ones. Using this approach, we found that a relatively small gain of 1.1× combined with an exponent of 3.8 led to a significantly larger effective gain of 1.37× after just one layer (Figure 3j). This form of non-linearity is consistent with the finding that static output non-linearities in single units range from about 2–4 (Gardner et al., 1999; Albrecht and Hamilton, 1982; Sclar et al., 1990; Heeger, 1992) and suggests a plausible physiological mechanism by which the larger gains predicted by our model could be implemented. Repeated use of exponentiation and normalization in successive layers of the visual system could produce an even larger effective gain. To avoid training a new convolutional neural network (CNN) and possibly violate the close relationship between the primate visual system and the CNN we studied, we continued our analysis without introducing an exponentiation and normalization step.

Figure 3

Download asset Open asset

The Gaussian gain could have its effect on the neural network observer model’s performance by increasing the activation strength of units with receptive fields near the locus of attention. These changes in activation strength might directly modify behavior, or work indirectly through mechanisms such as changes in receptive field size, location, or spatial tuning. We observed all of these effects in our model (Figure 3). To measure receptive fields we computed the derivative of each unit with respect to the input image and then fit these with a 2D Gaussian (see Materials and methods for details). We found that the gain caused receptive fields to shift and shrink toward the locus of attention (Figure 3a and b). The information provided by individual units in the model also changed, increasing for units on the border of the cued quadrant (Figure 3c). The receptive field shift and shrinkage were magnified in deeper layers of the model (Figure 3d and e) consistent with physiological observations (Klein et al., 2014). The gain in activation strength propagated through the network without modification (Figure 3f). To measure the effective gain experienced by the layer four units (Figure 3i), we computed the ratio of the standard deviations of unit activations at the output of each layer (Figure 2d) with and without gain applied. All three observed effects: receptive field shift, shrinkage and expansion, and effective gain were directly related to the gain strength at the input layer (Figure 3g–i). All of these changes have been proposed as mechanisms that could account for the behavioral benefits of attention (Anton-Erxleben and Carrasco, 2013; Moran and Desimone, 1985; Anton-Erxleben et al., 2009; Sprague and Serences, 2013; Vo et al., 2017; Kay et al., 2015; Fischer and Whitney, 2009; Anton-Erxleben et al., 2007). We designed models to try to isolate these effects with the goal of testing their independent contributions to task performance.

We next sought to test whether receptive field shifts alone could account for the behavioral benefits of the neural network observer model. To do this, we built a model variant that could shift receptive fields without introducing gain. To develop an intuition for how this could affect perceptual reports, consider a CNN with just four units in a 2 × 2 grid with each unit having its receptive field centered on one image in the composite. When shown a composite grid of four images, a logistic regression using the output of these four units would receive one quarter the information it expects from being trained on full size images. Shifting the receptive fields of the three non-target units to overlap more with the cued image could add additional task-relevant information to the output, much as was observed for units with receptive fields overlapping multiple images in the Gaussian gain attention model (Figure 3c).

We designed a variant of our model that could be used to test the hypothesis that receptive field shifts alone are responsible for the behavioral enhancement (Figure 4). In this model, we re-wired the units in the first layer to reproduce the effect of Gaussian gain. The re-wiring was designed so that receptive fields in the fourth layer matched their shift with the Gaussian gain model (Figure 3g). To mimic those shifts, we changed the connections between the input image pixels and layer one (Figure 4a). This manipulation worked as designed and changed the receptive field locations and size (Figure 4b–d) but since no gain was added to the model, the overall responsiveness of units remained constant (Figure 4e). Because receptive field shifts due to gain are not the result of actual rewiring it is unsurprising that the shift and shrinkage in this model variant are only qualitatively matched to those caused by the original Gaussian gain. Note that the effective gain of individual units in layer four did change for individual images, a result of each unit receiving different inputs, but the average change across images was zero.

Figure 4

Download asset Open asset

We found that the model with receptive field shifts but no gain had no effect on task performance, demonstrating that receptive field shifts are not key for the improvement in task performance observed with Gaussian gain (Figure 4f). The model imitating shifts from $4\times$ Gaussian gain had a median AUC across categories of 0.71, 95% CI [0.66, 0.73] compared to 0.71 [0.67, 0.72] in the baseline model, median change in AUC of –0.01 [-0.02, 0.01] within each category.

Another way to understand the possible effect of the Gaussian gain on task performance is to note that the spatial tuning profile of units is ‘shifted’ towards the locus of attention: sensitivity is enhanced closer to the locus of attention, but the receptive field itself has not truly moved in the manner studied by the previous model. If different parts of a receptive field receive asymmetric gain, as expected for Gaussian gain then the local structure of the receptive field has been changed (Figure 5a). We designed another model variant to test the hypothesis that these local changes in receptive field structure might be sufficient to explain changes in task performance without inducing receptive field shifts or gain. To implement this model at layer $L$, we examined the effect of the Gaussian gain on each unit (green differential gain, Figure 5a). We normalized this differential gain within each unit’s receptive field to prevent any overall gain effect and re-scaled the unit’s kernel accordingly. Overall, this manipulation of each unit’s kernel preserved a portion of the receptive field shift effect in a gain-dependent manner but guaranteed that there was no effective gain.

Figure 5

Download asset Open asset

The receptive field structure model was designed to only change the spatial tuning of individual units without inducing gain, which naturally caused some shifts in the measured receptive field size and location (solid lines and markers, Figure 5b–d) but these were smaller than the effects observed under Gaussian gain (dashed lines). The normalization prevented the model from introducing any spatial pattern of gain change (Figure 5e). Note that there were still small changes in overall sensitivity of units in this model, for example, the $4\times$ model had an average gain of 1.08, 95% CI [1.07, 1.09] across all units, which we attribute to the fact that inputs to a unit may exhibit correlations due to spatial structure. These receptive field changes and small gain effects were distinct from those observed under Gaussian gain (Figure 5c–e).

The receptive field structure model, like the shift model, was unable to account for the behavioral effects of the Gaussian gain. No matter where in the model we changed the receptive field structure, and even when applied at all layers, the average performance across categories remained flat (Figure 5f). Compared to the median distributed AUC across categories of 0.71 [0.67, 0.72], the sensitivity model applied to all layers had a median AUC across categories of 0.69 [0.65, 0.72] when imitating gain of $1.1\times$, 0.70 [0.65, 0.72] for $2\times$ gain, 0.69 [0.65, 0.71] for $4\times$ and 0.66 [0.63, 0.69] for $11\times$. Each of these conditions resulted in a median AUC change within category of –0.02 [-0.03, 0.00], –0.01 [-0.03, 0.00], -0.02 [-0.04,–0.01], and -0.04 [-0.05,–0.03], respectively. When applied to early layers, we observed a slight drop in performance, which we attribute to how this model directly alters the kernels in the CNN. These changes break the assumption that the CNN kernels at each layer are consistent with those that were optimized when the model weights were trained.

The Gaussian gain also caused units to shrink and expand their receptive fields across the visual field (Figure 3b). These changes might modify the information content received at the output layer, improving or hurting performance. We designed a model variant to test the hypothesis that shrinkage and expansion of receptive fields, without shift or gain, might be sufficient to explain the behavioral effect (Figure 6). To implement this model, we took the observed change in receptive field size at layer 4 and then re-scaled the connections between layers three and four to mimic the observed effect. Because the kernels were scaled in space this manipulation has no effect on effective gain or receptive field position. Specifically, we approximated the shrinkage of the sampled units using a parameterized equation (Equation 5) that provides a shrinkage factor for every unit in the model (Figure 6a). We then re-wired the connections between layer three and four using linear interpolation to approximate the necessary change in scaling.

Figure 6

Download asset Open asset

After re-wiring, units’ receptive fields retained the same overall position, but were scaled to qualitatively match the observed effects under Gaussian gain (Figure 6b–d). The size changes do not match perfectly with those under Gaussian gain because we enforced symmetry in two ways: first, by parameterizing the shrinkage and expansion we enforced symmetry around the locus of attention, and second, because the observed receptive field changes were often asymmetric but we implemented a symmetric linear scaling. These simplifications were necessary to reduce the complexity of implementation. By design, the shrinkage and expansion effects scaled by attention strength (Figure 6d) and induced neither gain-dependent shifts (Figure 6c) or effective gain (Figure 6e).

The shrinkage model was unable to account for improved task performance with Gaussian gain. The average performance across categories remained flat (Figure 6f). Compared to the median distributed AUC across categories of 0.71 [0.67, 0.72], the shrinkage model applied to all layers had a median AUC across categories of 0.70 [0.66, 0.72] when imitating gain of $1.1\times$, 0.70 [0.66, 0.71] for $2\times$ gain, 0.69 [0.65, 0.70] for $4\times$ and 0.68 [0.65, 0.70] for $11\times$. Each of these conditions resulted in a median AUC change within category of -0.01 [-0.01,–0.00], -0.01 [-0.01,–0.01], -0.02 [-0.02,–0.01], and –0.02 [–0.03, –0.01], respectively. We again observed drops in performance, which we attribute to how the kernels have been altered.

Having ruled out that receptive field shift, shrinkage, or changes in spatial tuning could account for the improved task performance in our neural network observer, we next designed a model to amplify signals in the cued quadrant without these other effects and found that this model was able to explain the improved task performance observed with cued attention. In the original Gaussian gain model, an asymmetry in gain was introduced in the receptive fields of the units, causing size and location changes in the receptive fields. To remove this effect, we flattened the gain within the cued quadrant (Figure 7a) by setting the gain at each pixel to the average of the Gaussian gain across the entire quadrant. By itself, this change has the unintended consequence that units partly overlapping the cued quadrant will still shift in a gain-dependent manner. To remove this effect, we split the CNN feature maps into the four quadrants and computed these separately with padding and concatenated the results. This forces all units in the model to receive information about only a single quadrant. These manipulations did result in shifts in receptive field location and size for units at the borders (Figure 7b–d), but by design these were independent of the gain strength.

Figure 7

Download asset Open asset

Using the gain-only model we were able to reproduce the improved task performance of the original Gaussian gain (Figure 7). The gain-only model induced the same pattern of receptive field shift and size change at all gain strengths (Figure 7b–d) and a flat effective gain within the cued quadrant (Figure 7e). We found that increasing the strength of a flat gain was sufficient to capture the full performance improvement of the original model (Figure 7f). The median AUC across categories of the $4\times$ flat gain model was 0.78, 95% CI [0.76, 0.83] compared to 0.80 [0.77, 0.82] for the $4\times$ Gaussian gain model. The confidence intervals in flat gain and Gaussian gain performance overlapped at all gain strengths, with a difference of 0.00 [-0.00, 0.02] at 1.1× gain, –0.01 [-0.02, 0.00] at 2× gain, –0.01 [-0.02, 0.00] at 4× gain, and 0.02 [0.00, 0.04] at 11× gain.

Having found that the improved task performance could be explained not by receptive field changes, but instead by the change in the overall gain, we asked whether gain propagated through the network was both necessary and sufficient to explain this effect. To test necessity and sufficiency, we ran the task images through the Gaussian gain model (first row, Figure 8a) and measured the effective gain propagated to units in the final layer output (7 × 7 × 512, before averaging). We averaged these effective gains over features to obtain a propagated gain map (Layer 4 feature map, 7 × 7, Figure 8b). To test the hypothesis that this propagated gain was sufficient to account for improved performance in the task, we re-applied it to the output layer of a model with no gain applied to the inputs.

Figure 8 with 2 supplements see all

Download asset Open asset

We found that the propagated gain map, when used to multiply the outputs of a model with no Gaussian gain (Multiply by propagated gain, Figure 8a) was sufficient to induce task performance benefits similar to Gaussian gain applied to the input (Propagated gain vs. Gaussian gain, Figure 8c). The median AUC across categories using the propagated gain map was 0.79, 95% CI [0.76, 0.84], compared to 0.71 [0.67, 0.72] in the distributed model. There was a small difference between the Gaussian gain and the effect of the propagated gain map -0.02 [–0.03, 0.01], within the 95% confidence interval for no difference. This difference could be attributed to changes in receptive field structure in the Gaussian gain condition, but we attribute it instead to differences between the propagated gain map and the effect of the Gaussian gain. The propagated gain manipulation was constructed from the average effective gain of units across all task stimuli. Because of this, the gain map did not exactly reproduce the effect of gain on an image-by-image basis.

To test the hypothesis that gain was necessary to account for the behavioral effect, we divided the final layer activations by the propagated gain map (Divide by propagated gain, Figure 8a). We found that the behavioral effect of an early gain was mostly reversed by this manipulation (Removed gain vs. Distributed, Figure 8c). The median AUC across categories after dividing out the propagated gain was 0.72, 95% CI [0.68, 0.75], compared to 0.71 [0.67, 0.72] in the distributed condition. Dividing by the propagated map did not perfectly reverse the effects of the full Gaussian gain. When compared to the distributed baseline, we found a median within category AUC advantage for Gaussian gain with division of 0.02 [0.01, 0.03]. These small residual differences are likely due to the combined effects of changes in spatial receptive field properties that are not reversed by the division of the propagated gain map.

Note that in the final readout of our model, we assumed that explicit spatial information was lost, as we averaged activations across the $7\times 7$ convolutional units in the final pooling layer. However, some evidence in ventral temporal cortex suggests that there is spatial information available (Schwarzlose et al., 2008; Carlson et al., 2011), so we tested a model read out which retained spatial information and found that the necessary and sufficiency results did not show qualitative changes. A model trained to use the full $7\times 7\times 512$ output had marginally worse performance than the model built with the average encodings, achieving a median AUC across categories of 0.68 [0.63, 0.71] in the distributed condition and 0.78 [0.75, 0.82] in the focal condition with $4\times$ Gaussian gain at the first layer. We attribute the small difference in task performance compared to the average model to worse generalization: on the validation set the $7\times 7$ model showed a median drop in AUC across categories of –0.02, range [–0.10, 0.00] compared to the average-pooled readouts.

We repeated the propagated gain manipulations in the $7\times 7$ readout model to confirm the necessary and sufficiency results would not change when the model retained spatial information in the final readout. Both the necessity and sufficiency tests showed similar results when using the full output: the average increase in AUC when using the propagated gain map was 0.09, 95% CI [0.08, 0.10] for the full output model, compared to 0.09 [0.07, 0.10] for the average pooled model and the average change in AUC (compared to the distributed condition) when dividing out the propagated gain map from a model with Gaussian gain applied was 0.01 [-0.01, 0.02] for the full output model, compared to 0.02 [0.01, 0.03] for the average pooled model.

Improvements in task performance with a Gaussian gain could come from changes in signal discriminability, but also could come from the network being better able to suppress irrelevant visual information. That is, increasing the gain could act to strengthen signals from the relevant target and suppress signals from irrelevant locations. To see how much suppressing irrelevant visual information alone could improve task performance, we designed a neural network observer model which explicitly read out from the top-left $4\times 4\times 512$ quadrant of the layer 4 output, instead of the average pooled $1\times 1\times 512$ output. As expected, the task performance of this model with no additional gain is already elevated (Distributed, Figure 8—figure supplement 1), because the readout now implicitly acts as a form of spatial cueing. The performance of the 4×4 readout was still not at ceiling (ΔAUC between training validation set and distributed images = –0.07, 95% CI [-0.06,–0.09]). Thus, the performance enhancement due to the Gaussian gain appears to act similarly to an explicit manipulation which suppresses irrelevant information.

Theoretical considerations would suggest that moving receptive fields into the target quadrant should further improve performance even when the readout is already spatially specific, because these additional receptive fields can add new information (Kay et al., 2015; Vo et al., 2017). We found that this was not true for the amount of shift induced by the 4× Gaussian gain, chosen to match the magnitude of the human behavioral benefits of spatial attention. To demonstrate this, we applied a 4× Gaussian gain to the $4\times 4$ readout model and found no further increase in performance beyond what was achieved by shifting the readout (Gaussian gain vs. 4 × 4 Readout, Figure 8—figure supplement 1). Gain applied to the output of the model also provided no additional benefit (Propagated Gain vs. 4 × 4 Readout, Figure 8—figure supplement 1), supporting the interpretation that the gain acts as a selection mechanism with no effect in the absence of irrelevant distractors.

Finally, the observer model solved a detection task where both criterion and sensitivity contribute to performance and we reported task performance as AUC to avoid confounding these factors. A more explicit test is to use a criterion-free discrimination task to evaluate the effects of gain on task performance. (Figure 8—figure supplement 2a). We therefore designed a category discrimination task in which the neural network observer model determined which of two composite grids included the target category at a specified location (always top-left). Baseline performance (Baseline, Figure 8—figure supplement 2b) was considered as the performance of the model when no information about which location is cued for discrimination was provided. Note that because the discrimination location always included the target and all other locations had equal probability of including the target category, chance performance was greater than 50%. To compute task performance, the previously trained fully connected category target readout was compared across the two composite grids and the composite with the larger response chosen as the model’s response. We then applied a Gaussian gain at the cued location and found that discrimination task performance improved in a similar manner to the detection task (Gaussian Gain, Figure 8—figure supplement 2b). Using the propagated gain manipulation, we confirmed that the gain was both necessary and sufficient for improvements in model task performance (Removed Gain vs. Baseline and Propagated Gain vs. Gaussian Gain, Figure 8—figure supplement 2b).

Human observers are more accurate when trying to detect or discriminate objects at a cued location. Our results demonstrate that this behavioral benefit can also be observed in a neural network model of visual cortex when a Gaussian gain is applied over the pixels of a ‘cued’ object. By modeling attentional modulation as gain at the earliest stage of the neural network, we were able to observe similar effects on spatial receptive fields to what is seen in human physiology. When using a gain strength set to match model and human performance enhancement during spatial attention, we documented shifts of receptive fields towards the center of the Gaussian gain field, shrinkage of receptive fields, and changes to the spatial structure in units at later stages of the model. These changes in model receptive field properties were similar in magnitude and characteristics to changes in single-unit (Womelsdorf et al., 2006; Anton-Erxleben et al., 2009) and population (Klein et al., 2014; Vo et al., 2017; Fischer and Whitney, 2009; van Es et al., 2018) receptive fields reported from physiological measurements.

To determine which, if any, of these changes to receptive field properties were the source of improved task performance in the model, we built a series of neural network observer models in which we isolated receptive field shifts, shrinkage, and structural changes from the direct effect of gain. To assess these changes in a way that could provide information about the human visual system, we matched the scale of the shifts, shrinkage, and structural changes to the effect size observed in the Gaussian gain model with the gain strength best matched to human performance. In the shift-only model, we re-wired units to move receptive fields without introducing gain and found that this produced no improvements in task performance. In the shrinkage model, we changed the size of units without changing their gain or position, and again found no improvements in task performance. In the receptive field structure model, we modified the sensitivity profile of individual receptive fields to mimic the effects of gain, without changing their gain, position, or size, but again found no improvements in task performance. It was only by applying a gain while keeping receptive field properties stable that we were able to reproduce the improvements in task performance.

Our results suggest that spatial gain implemented by neural populations in visual cortex can be sufficient to induce behavioral effects of attention for both detection and discrimination even without the concomitant changes in downstream receptive field properties. That is, increasing response magnitudes through gain changes can act to select relevant visual information when coupled with a max or soft-max pooling to suppress irrelevant visual information which has a lower response magnitude (Lee et al., 1999; Pestilli et al., 2011; Hara et al., 2014; Pelli, 1985). Although increasing gain can have downstream effects on receptive field properties such as changes in position, size and spatial structure, our results suggest that these may be secondary effects and only a consequence of applying gain, rather than the cause of the behavioral improvements as others have suggested (Anton-Erxleben and Carrasco, 2013; Moran and Desimone, 1985; Anton-Erxleben et al., 2009; Sprague and Serences, 2013; Vo et al., 2017; Kay et al., 2015; Fischer and Whitney, 2009; Anton-Erxleben et al., 2007). In addition, we found that gain had no additional impact when the readout was already spatially specific, reinforcing the interpretation that gain and selection of relevant information are intertwined.

We used an image-computable model of the computational steps from sensory input to decision making which allowed us to formally test hypotheses (Gardner and Merriam, 2021) about how different attentional mechanisms could impact task performance. In our case, the advantage of this approach is that the model architecture allowed us to examine how gain at the earliest stages of processing causes changes in spatial receptive field properties: any time a gain occurs in an asymmetrical manner across a receptive field, downstream units will experience an apparent shift as well as shrinkage or expansion. We know from the large literature exploring the physiology of attention that receptive field shifts are correlated with spatial attention (Anton-Erxleben and Carrasco, 2013; Anton-Erxleben et al., 2009; Anton-Erxleben et al., 2007; Vo et al., 2017; Kay et al., 2015; Fischer and Whitney, 2009; Womelsdorf et al., 2006). Several authors have proposed that enhanced task performance is a result of increasing information capacity by reducing spatial uncertainty about position (Kay et al., 2015) or enhancing discriminability (Vo et al., 2017). However, if changes in spatial receptive field properties are the consequence of gain changes (Klein et al., 2014; Compte and Wang, 2006), then it raises the question of whether these receptive field changes actually help to improve task performance. Our modeling approach allowed us to examine the theoretical impact of each change associated with gain systematically and quantify that these provided no benefit to detection or discrimination task performance. Nevertheless, previous modeling work has demonstrated that shift and shrinkage, in particular, can increase the resolution and redundancy of receptive field coverage (Kay et al., 2015; Vo et al., 2017; Theiss et al., 2022). Our results differ along two important dimensions: first, we scaled the magnitude of shift, shrinkage, and sensitivity changes to those induced by Gaussian gain (at a gain strength that was matched to human performance) and these were in general equal to or smaller in magnitude to what was observed in these previous papers. This leaves open the possibility that larger shifts or shrinkage could have a more direct impact on task performance. We also note that our results were based only on simulations for category detection and discrimination tasks. It may be the case that other tasks which depend more directly on spatial coding, such as judgements of visual position, could exhibit larger benefits from shifts and shrinkage of receptive fields (Kay et al., 2015; Vo et al., 2017).

Whether our conclusions can generalize to the behavior of attentional gain in biological neural circuits is limited both by how well the neural network observer model approximates the functioning of those neural circuits and by the model’s ability to predict behavior. There are several reasons to suggest that the model captures relevant properties of both object recognition and the primate visual system. We chose to analyze a CNN whose architecture was designed to reflect the primate visual system. This has been evaluated by comparing the similarity of CNN unit activity against measurements of single unit activity in the primate visual cortex (Schrimpf et al., 2018). After training, the image features that the CNN units become selective for align closely with those that activate single units in visual cortex (Yamins et al., 2014; Carter et al., 2019). In addition, the designers of the architecture we used (CORnet), Kubilius et al., 2018 optimized for ‘core object recognition’, detecting a dominant object during a viewing duration of natural fixation (100–200ms) in the central visual field (10 deg). We re-used core object recognition in our human object detection task and projected our composites in a 10 degree square aperture to obtain similar perceptual characteristics. In the analysis of our task we showed that distributed performance was similar for humans and the CNN at a stimulus presentation of 65ms, confirming that the intended design of CORnet generalized to the new dataset and task that we used.

While CORnet was designed to map individual visual cortex regions onto the different layers of the CNN, it differs from the visual system in that it is a completely feed-forward model. It is well-known that the visual system has recurrence both within and between visual areas (Felleman and Van Essen, 1991). Computational modeling has suggested that recurrence can affect how gain and additive offsets change down stream receptive field location and size, in particular enhancing these effects beyond receptive field boundaries (Compte and Wang, 2006). These considerations suggest that more realistic models could have even stronger downstream effects on spatial receptive field properties then what we have documented in a purely feed-forward network. In computational models, recurrent connections are often unfolded into feed-forward layers, effectively making a recurrent model a deeper convolutional model (Nayebi et al., 2018). Although we did not test deeper architectures in our analysis, we expect that the general principles we described should hold for models with more layers and therefore also for models with recurrent connections. An intriguing follow-up direction would be to extend the modeling described here to reaction time tasks, where a recurrent architecture allows for modeling of temporal dynamics and where diffusion models have been found to provide a useful parameterization of how bottom-up and top-down signals contribute to sensory responses over time (Kay and Yeatman, 2017).

CORnet is also missing many intermediate areas of the visual system (notably area V3) (Wandell and Winawer, 2011) as well as an explicit gain control mechanism such as divisive normalization (Carandini and Heeger, 2012) which might account for the large gain necessary in our model to produce human-like performance enhancements. These differences mean that the exact strength of the gain signal we observed cannot be mapped directly onto physiology. In particular, while we apply gain at the earliest stage of the model, we do not wish to imply that such a large gain is seen with attention in the LGN inputs to V1 (O’Connor et al., 2002). Nor do we imply that the gain in various stages of our model should directly map on to the gain observed in physiological measurements, which have tended to highlight larger gain changes in intermediate areas like V4 and MT (Treue and Martínez Trujillo, 1999; McAdams and Maunsell, 1999; Moore and Armstrong, 2003) compared to earlier areas. Instead, in our model, the 4× gain should be interpreted as both an explicit increase in gain as well as an implicit gain due to the effects of normalization (Reynolds and Heeger, 2009; Carandini and Heeger, 2012). While normalization models have traditionally been studied in single layer models, our work extends this general approach to consider downstream effects of gain on RF properties. We assessed how these effects might interact in our CNN by demonstrating that a physiologically plausible gain of 1.1×, when accentuated by a divisive gain control mechanism (Kaiser et al., 2016; Carandini and Heeger, 2012) and amplified across multiple visual areas (many of which are not included in the CORnet model), could have produced the magnitude of effects necessary for human-level improvements in task performance. This smaller gain is more consistent with neural recordings in primates, where gain changes on the order 20–40% (1.2–1.4×) have been measured (Motter, 1993; Luck et al., 1997; Treue and Martínez Trujillo, 1999).

We chose to model gain at the earliest possible point in the system to understand how signal changes propagate through the visual hierarchy and modify receptive field structure. Physiological measurements have found evidence for early gain (McAdams and Maunsell, 1999; Motter, 1993; Luck et al., 1997), but it is equally possible that the gain is applied at a late stage close to decision making and signal gains early in visual cortex are a result of backward projections to these areas (Buffalo et al., 2010; Moore and Armstrong, 2003). The propagated gain analysis confirms that gain signals with spatial specificity arriving at later stages in processing (Moore and Armstrong, 2003) would have similar effects on task performance.

To solve the demands of goal-directed visual attention, the human brain has multiple potential mechanisms available. To select for relevant and suppress irrelevant information, sensory responses can be amplified or the tuning of neurons and populations can be shifted to enhance some signals at the cost of others. In addition, these bottom-up sensory changes can be combined with changes in how sensory representations are read out or communicated to downstream regions. In biological systems, these mechanisms are intertwined: as we have shown, changes to early sensory signals will have complex effects on the later stages that are used for readout. In an idealized model, the changes that would have the most effect on the readout would be computed by approximating their gradients on the decision axis (Lindsay and Miller, 2018). However, these gradients are typically computed in models through back-propagation (Rumelhart et al., 1986), and it is not known whether or how similar gradients can be computed in biological systems. Here, we have shown using a state-of-the-art model of the visual system that when the neural network observer is matched with human performance during spatial attention some mechanisms can improve task performance, while others cannot. In the limit, shift, shrinkage, and tuning changes in receptive fields must have an impact on sensory representations and therefore on performance. But our results show that in a neural network model and at the scale expected in the primate visual system during goal-directed behavior, these are not sufficient to produce the expected effects of spatial attention on task performance. Instead, gain combined with a nonlinear selection mechanism meets the demands imposed by goal-directed visual attention. New techniques that allow for targeting interventions to defined populations of neurons raise the possibility of manipulating gain and top-down signaling to determine the effect on downstream neural response properties and behavior. Such interventions would allow for testing the main prediction of our model: that spatial visual attention relies primarily on changes in gain and not concomitant downstream effects to spatial receptive field properties.

The images and composite grids used in this study as well as the code necessary to replicate our analyses are available in the Open Science Framework with the identifier https://doi.org/10.17605/OSF.IO/AGHQK.

1. 1. Fox KJ
   2. Birman D
   3. Gardner JL

   (2020) Open Science Framework

   Gain, not changes in spatial receptive field properties, improves task performance in a neural network attention model.

   https://doi.org/10.17605/OSF.IO/AGHQK

1. 1. Albrecht DG
   2. Hamilton DB

   (1982) Striate cortex of monkey and cat: contrast response function

   Journal of Neurophysiology 48:217–237.

   https://doi.org/10.1152/jn.1982.48.1.217

   • PubMed
   • Google Scholar
2. 1. Anton-Erxleben K
   2. Henrich C
   3. Treue S

   (2007) Attention changes perceived size of moving visual patterns

   Journal of Vision 7:5.

   https://doi.org/10.1167/7.11.5

   • PubMed
   • Google Scholar
3. 1. Anton-Erxleben K
   2. Stephan VM
   3. Treue S

   (2009) Attention reshapes center-surround receptive field structure in Macaque cortical area mt

   Cerebral Cortex 19:2466–2478.

   https://doi.org/10.1093/cercor/bhp002

   • PubMed
   • Google Scholar
4. 1. Anton-Erxleben K
   2. Carrasco M

   (2013) Attentional enhancement of spatial resolution: linking behavioural and neurophysiological evidence

   Nature Reviews. Neuroscience 14:188–200.

   https://doi.org/10.1038/nrn3443

   • PubMed
   • Google Scholar
5. 1. Ben Hamed S
   2. Duhamel JR
   3. Bremmer F
   4. Graf W

   (2002) Visual receptive field modulation in the lateral Intraparietal area during attentive fixation and free gaze

   Cerebral Cortex 12:234–245.

   https://doi.org/10.1093/cercor/12.3.234

   • PubMed
   • Google Scholar
6. 1. Birman D
   2. Gardner JL

   (2019) A flexible Readout mechanism of human sensory representations

   Nature Communications 10:3500.

   https://doi.org/10.1038/s41467-019-11448-7

   • PubMed
   • Google Scholar
7. 1. Buffalo EA
   2. Fries P
   3. Landman R
   4. Liang H
   5. Desimone R

   (2010) A backward progression of Attentional effects in the ventral stream

   PNAS 107:361–365.

   https://doi.org/10.1073/pnas.0907658106

   • Google Scholar
8. 1. Cadena SA
   2. Denfield GH
   3. Walker EY
   4. Gatys LA
   5. Tolias AS
   6. Bethge M
   7. Ecker AS

   (2019) Deep Convolutional models improve predictions of Macaque V1 responses to natural images

   PLOS Computational Biology 15:e1006897.

   https://doi.org/10.1371/journal.pcbi.1006897

   • PubMed
   • Google Scholar
9. 1. Carandini M
   2. Heeger DJ

   (2012) Normalization as a Canonical neural computation

   Nature Reviews Neuroscience 13:51–62.

   https://doi.org/10.1038/nrn3136

   • Google Scholar
10. 1. Carlson T
    2. Hogendoorn H
    3. Fonteijn H
    4. Verstraten FAJ

    (2011) Spatial coding and Invariance in object-selective cortex

    Cortex; a Journal Devoted to the Study of the Nervous System and Behavior 47:14–22.

    https://doi.org/10.1016/j.cortex.2009.08.015

    • PubMed
    • Google Scholar
11. 1. Carrasco M

    (2011) Visual attention: the past 25 years

    Vision Research 51:1484–1525.

    https://doi.org/10.1016/j.visres.2011.04.012

    • PubMed
    • Google Scholar
12. 1. Carter S
    2. Armstrong Z
    3. Schubert L
    4. Johnson I
    5. Olah C

    (2019) Activation Atlas

    Distill 4:e15.

    https://doi.org/10.23915/distill.00015

    • Google Scholar
13. 1. Cichy RM
    2. Khosla A
    3. Pantazis D
    4. Torralba A
    5. Oliva A

    (2016) Comparison of deep neural networks to Spatio-temporal cortical Dynamics of human visual object recognition reveals Hierarchical correspondence

    Scientific Reports 6:27755.

    https://doi.org/10.1038/srep27755

    • PubMed
    • Google Scholar
14. 1. Cohen MR
    2. Newsome WT

    (2008) Context-dependent changes in functional circuitry in visual area mt

    Neuron 60:162–173.

    https://doi.org/10.1016/j.neuron.2008.08.007

    • PubMed
    • Google Scholar
15. 1. Colby CL
    2. Goldberg ME

    (1999) Space and attention in Parietal cortex

    Annual Review of Neuroscience 22:319–349.

    https://doi.org/10.1146/annurev.neuro.22.1.319

    • PubMed
    • Google Scholar
16. 1. Compte A
    2. Wang XJ

    (2006) Tuning curve shift by attention modulation in cortical neurons: a computational study of its mechanisms

    Cerebral Cortex 16:761–778.

    https://doi.org/10.1093/cercor/bhj021

    • PubMed
    • Google Scholar
17. 1. Connor CE
    2. Gallant JL
    3. Preddie DC
    4. Van Essen DC

    (1996) Responses in area V4 depend on the spatial relationship between stimulus and attention

    Journal of Neurophysiology 75:1306–1308.

    https://doi.org/10.1152/jn.1996.75.3.1306

    • PubMed
    • Google Scholar
18. Conference

    1. Deng J
    2. Dong W
    3. Socher R
    4. Li LJ

    (2009) ImageNet: A large-scale hierarchical image database

    2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops.

    https://doi.org/10.1109/CVPR.2009.5206848

    • Google Scholar
19. 1. Duhamel JR
    2. Colby CL
    3. Goldberg ME

    (1992) The updating of the representation of visual space in Parietal cortex by intended eye movements

    Science 255:90–92.

    https://doi.org/10.1126/science.1553535

    • PubMed
    • Google Scholar
20. 1. Eckstein MP
    2. Thomas JP
    3. Palmer J
    4. Shimozaki SS

    (2000) A signal detection model predicts the effects of set size on visual search accuracy for feature, conjunction, triple conjunction, and Disjunction displays

    Perception & Psychophysics 62:425–451.

    https://doi.org/10.3758/BF03212096

    • Google Scholar
21. 1. Eickenberg M
    2. Gramfort A
    3. Varoquaux G
    4. Thirion B

    (2017) Seeing it all: Convolutional network layers map the function of the human visual system

    NeuroImage 152:184–194.

    https://doi.org/10.1016/j.neuroimage.2016.10.001

    • PubMed
    • Google Scholar
22. 1. Felleman DJ
    2. Van Essen DC

    (1991) Distributed Hierarchical processing in the Primate cerebral cortex

    Cerebral Cortex 1:1–47.

    https://doi.org/10.1093/cercor/1.1.1-a

    • PubMed
    • Google Scholar
23. 1. Fischer J
    2. Whitney D

    (2009) Attention NARROWS position tuning of population responses in V1

    Current Biology 19:1356–1361.

    https://doi.org/10.1016/j.cub.2009.06.059

    • PubMed
    • Google Scholar
24. 1. Gardner JL
    2. Anzai A
    3. Ohzawa I
    4. Freeman RD

    (1999) Linear and Nonlinear contributions to orientation tuning of simple cells in the cat’s Striate cortex

    Visual Neuroscience 16:1115–1121.

    https://doi.org/10.1017/s0952523899166112

    • PubMed
    • Google Scholar
25. Software

    1. Gardner JL
    2. Merriam EP
    3. Schluppeck D
    4. Larsson J

    (2018) MGL: visual Psychophysics stimuli and experimental design package, version 2.0

    Zenodo.

    https://doi.org/10.5281/zenodo.1299497
26. 1. Gardner JL
    2. Merriam EP

    (2021) Population models, not analyses, of human Neuroscience measurements

    Annual Review of Vision Science 7:225–255.

    https://doi.org/10.1146/annurev-vision-093019-111124

    • PubMed
    • Google Scholar
27. 1. Güçlü U
    2. van Gerven MAJ

    (2015) Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream

    The Journal of Neuroscience 35:10005–10014.

    https://doi.org/10.1523/JNEUROSCI.5023-14.2015

    • PubMed
    • Google Scholar
28. 1. Hara Y
    2. Pestilli F
    3. Gardner JL

    (2014) Differing effects of attention in single-units and populations are well predicted by heterogeneous tuning and the normalization model of attention

    Frontiers in Computational Neuroscience 8:12.

    https://doi.org/10.3389/fncom.2014.00012

    • PubMed
    • Google Scholar
29. 1. Hawkins HL
    2. Hillyard SA
    3. Luck SJ
    4. Mouloua M
    5. Downing CJ
    6. Woodward DP

    (1990) Visual attention modulates signal Detectability

    Journal of Experimental Psychology. Human Perception and Performance 16:802–811.

    https://doi.org/10.1037//0096-1523.16.4.802

    • PubMed
    • Google Scholar
30. 1. Heeger DJ

    (1992) Half-squaring in responses of cat Striate cells

    Visual Neuroscience 9:427–443.

    https://doi.org/10.1017/S095252380001124X

    • Google Scholar
31. 1. Kaiser D
    2. Oosterhof NN
    3. Peelen MV

    (2016) The neural Dynamics of Attentional selection in natural scenes

    The Journal of Neuroscience 36:10522–10528.

    https://doi.org/10.1523/JNEUROSCI.1385-16.2016

    • PubMed
    • Google Scholar
32. 1. Kay KN
    2. Weiner KS
    3. Grill-Spector K

    (2015) Attention reduces spatial uncertainty in human ventral temporal cortex

    Current Biology 25:595–600.

    https://doi.org/10.1016/j.cub.2014.12.050

    • PubMed
    • Google Scholar
33. 1. Kay KN
    2. Yeatman JD

    (2017) Bottom-up and top-down computations in word-and face-selective cortex

    eLife 6:e22341.

    https://doi.org/10.7554/eLife.22341

    • PubMed
    • Google Scholar
34. 1. Khaligh-Razavi SM
    2. Kriegeskorte N

    (2014) Deep supervised, but not Unsupervised, models may explain it cortical representation

    PLOS Computational Biology 10:e1003915.

    https://doi.org/10.1371/journal.pcbi.1003915

    • PubMed
    • Google Scholar
35. 1. Klein BP
    2. Harvey BM
    3. Dumoulin SO

    (2014) Attraction of position preference by spatial attention throughout human visual cortex

    Neuron 84:227–237.

    https://doi.org/10.1016/j.neuron.2014.08.047

    • PubMed
    • Google Scholar
36. Conference

    1. Krizhevsky A
    2. Sutskever I
    3. Hinton GE

    (2012) Imagenet classification with deep Convolutional neural networks

    Advances in Neural Information Processing Systems. pp. 84–90.

    https://doi.org/10.1145/3065386

    • Google Scholar
37. Preprint

    1. Kubilius J
    2. Schrimpf M
    3. Nayebi A
    4. Bear D
    5. Yamins DLK
    6. DiCarlo JJ

    (2018) CORnet: Modeling the Neural Mechanisms of Core Object Recognition

    bioRxiv.

    https://doi.org/10.1101/408385

    • Google Scholar
38. 1. Kusunoki M
    2. Goldberg ME

    (2003) The time course of Perisaccadic receptive field shifts in the lateral Intraparietal area of the monkey

    Journal of Neurophysiology 89:1519–1527.

    https://doi.org/10.1152/jn.00519.2002

    • PubMed
    • Google Scholar
39. 1. Lee DK
    2. Itti L
    3. Koch C
    4. Braun J

    (1999) Attention activates winner-take-all competition among visual filters

    Nature Neuroscience 2:375–381.

    https://doi.org/10.1038/7286

    • PubMed
    • Google Scholar
40. 1. Lindsay GW
    2. Miller KD

    (2018) How biological attention mechanisms improve task performance in a large-scale visual system model

    eLife 7:e38105.

    https://doi.org/10.7554/eLife.38105

    • PubMed
    • Google Scholar
41. 1. Luck SJ
    2. Chelazzi L
    3. Hillyard SA
    4. Desimone R

    (1997) Neural mechanisms of spatial selective attention in areas V1, V2, and V4 of Macaque visual cortex

    Journal of Neurophysiology 77:24–42.

    https://doi.org/10.1152/jn.1997.77.1.24

    • PubMed
    • Google Scholar
42. 1. McAdams CJ
    2. Maunsell JH

    (1999) Effects of attention on orientation-tuning functions of single neurons in Macaque cortical area V4

    The Journal of Neuroscience 19:431–441.

    https://doi.org/10.1523/JNEUROSCI.19-01-00431.1999

    • PubMed
    • Google Scholar
43. Conference

    1. McIntosh LT
    2. Maheswaranathan N
    3. Nayebi A
    4. Ganguli S
    5. Baccus SA

    (2016)

    Deep learning models of the retinal response to natural scenes

    Advances in Neural Information Processing Systems. pp. 1369–1377.

    • Google Scholar
44. 1. Merriam EP
    2. Genovese CR
    3. Colby CL

    (2007) Remapping in human visual cortex

    Journal of Neurophysiology 97:1738–1755.

    https://doi.org/10.1152/jn.00189.2006

    • PubMed
    • Google Scholar
45. 1. Moore T
    2. Armstrong KM

    (2003) Selective gating of visual signals by Microstimulation of frontal cortex

    Nature 421:370–373.

    https://doi.org/10.1038/nature01341

    • PubMed
    • Google Scholar
46. 1. Moran J
    2. Desimone R

    (1985) Selective attention gates visual processing in the Extrastriate cortex

    Science 229:782–784.

    https://doi.org/10.1126/science.4023713

    • PubMed
    • Google Scholar
47. 1. Motter BC

    (1993) Focal attention produces spatially selective processing in visual cortical areas V1, V2, and V4 in the presence of competing stimuli

    Journal of Neurophysiology 70:909–919.

    https://doi.org/10.1152/jn.1993.70.3.909

    • PubMed
    • Google Scholar
48. Conference

    1. Nayebi A
    2. Bear D
    3. Kubilius J
    4. Kar K
    5. Ganguli S
    6. Sussillo D
    7. DiCarlo JJ
    8. Yamins DL

    (2018)

    Task-driven Convolutional recurrent models of the visual system

    Advances in Neural Information Processing Systems. pp. 5290–5301.

    • Google Scholar
49. 1. O’Connor DH
    2. Fukui MM
    3. Pinsk MA
    4. Kastner S

    (2002) Attention modulates responses in the human lateral Geniculate nucleus

    Nature Neuroscience 5:1203–1209.

    https://doi.org/10.1038/nn957

    • PubMed
    • Google Scholar
50. 1. Palmer J
    2. Verghese P
    3. Pavel M

    (2000) The Psychophysics of visual search

    Vision Research 40:1227–1268.

    https://doi.org/10.1016/S0042-6989(99)00244-8

    • Google Scholar
51. 1. Pedregosa F
    2. Varoquaux G
    3. Gramfort A
    4. Michel V
    5. Thirion B
    6. Grisel O
    7. Blondel M
    8. Prettenhofer P
    9. Weiss R
    10. Dubourg V

    (2011)

    Scikit-learn: machine learning in python

    The Journal of Machine Learning Research 12:2825–2830.

    • Google Scholar
52. 1. Pelli DG

    (1985) Uncertainty explains many aspects of visual contrast detection and discrimination

    Journal of the Optical Society of America A 2:1508.

    https://doi.org/10.1364/JOSAA.2.001508

    • Google Scholar
53. 1. Pestilli F
    2. Carrasco M
    3. Heeger DJ
    4. Gardner JL

    (2011) Attentional Enhancement via selection and pooling of early sensory responses in human visual cortex

    Neuron 72:832–846.

    https://doi.org/10.1016/j.neuron.2011.09.025

    • PubMed
    • Google Scholar
54. 1. Posner MI

    (1980) Orienting of attention

    The Quarterly Journal of Experimental Psychology 32:3–25.

    https://doi.org/10.1080/00335558008248231

    • PubMed
    • Google Scholar
55. 1. Reynolds JH
    2. Heeger DJ

    (2009) The normalization model of attention

    Neuron 61:168–185.

    https://doi.org/10.1016/j.neuron.2009.01.002

    • Google Scholar
56. 1. Ross J
    2. Morrone MC
    3. Burr DC

    (1997) Compression of visual space before Saccades

    Nature 386:598–601.

    https://doi.org/10.1038/386598a0

    • PubMed
    • Google Scholar
57. 1. Rumelhart DE
    2. Hinton GE
    3. Williams RJ

    (1986) Learning representations by back-Propagating errors

    Nature 323:533–536.

    https://doi.org/10.1038/323533a0

    • Google Scholar
58. 1. Sagi D
    2. Julesz B

    (1986) Enhanced detection in the aperture of focal attention during simple discrimination tasks

    Nature 321:693–695.

    https://doi.org/10.1038/321693a0

    • PubMed
    • Google Scholar
59. Preprint

    1. Schrimpf M
    2. Kubilius J
    3. Hong H
    4. Majaj NJ
    5. Rajalingham R
    6. Issa EB
    7. Kar K
    8. Bashivan P
    9. Prescott-Roy J
    10. Geiger F
    11. Schmidt K
    12. Yamins DLK
    13. DiCarlo JJ

    (2018) Brain-Score: Which Artificial Neural Network for Object Recognition Is Most Brain-Like?

    bioRxiv.

    https://doi.org/10.1101/407007

    • Google Scholar
60. 1. Schwarzlose RF
    2. Swisher JD
    3. Dang S
    4. Kanwisher N

    (2008) The distribution of category and location information across object-selective regions in human visual cortex

    PNAS 105:4447–4452.

    https://doi.org/10.1073/pnas.0800431105

    • Google Scholar
61. 1. Sclar G
    2. Maunsell JH
    3. Lennie P

    (1990) Coding of image contrast in central visual pathways of the Macaque monkey

    Vision Research 30:1–10.

    https://doi.org/10.1016/0042-6989(90)90123-3

    • PubMed
    • Google Scholar
62. 1. Sprague TC
    2. Serences JT

    (2013) Attention modulates spatial priority maps in the human occipital, Parietal and frontal Cortices

    Nature Neuroscience 16:1879–1887.

    https://doi.org/10.1038/nn.3574

    • PubMed
    • Google Scholar
63. Preprint

    1. Storrs KR
    2. Kietzmann TC
    3. Walther A
    4. Mehrer J
    5. Kriegeskorte N

    (2020) Diverse Deep Neural Networks All Predict Human IT Well, after Training and Fitting

    bioRxiv.

    https://doi.org/10.1101/2020.05.07.082743

    • Google Scholar
64. 1. Theiss JD
    2. Bowen JD
    3. Silver MA

    (2022) Spatial attention enhances crowded stimulus Encoding across modeled receptive fields by increasing redundancy of feature representations

    Neural Computation 34:190–218.

    https://doi.org/10.1162/neco_a_01447

    • Google Scholar
65. 1. Tolias AS
    2. Moore T
    3. Smirnakis SM
    4. Tehovnik EJ
    5. Siapas AG
    6. Schiller PH

    (2001) Eye movements modulate visual receptive fields of V4 neurons

    Neuron 29:757–767.

    https://doi.org/10.1016/s0896-6273(01)00250-1

    • PubMed
    • Google Scholar
66. 1. Treue S
    2. Martínez Trujillo JC

    (1999) Feature-based attention influences motion processing gain in Macaque visual cortex

    Nature 399:575–579.

    https://doi.org/10.1038/21176

    • PubMed
    • Google Scholar
67. 1. van Es DM
    2. Theeuwes J
    3. Knapen T

    (2018) Spatial sampling in human visual cortex is modulated by both spatial and feature-based attention

    eLife 7:e36928.

    https://doi.org/10.7554/eLife.36928

    • PubMed
    • Google Scholar
68. 1. Vo VA
    2. Sprague TC
    3. Serences JT

    (2017) Spatial tuning shifts increase the Discriminability and Fidelity of population codes in visual cortex

    The Journal of Neuroscience 37:3386–3401.

    https://doi.org/10.1523/JNEUROSCI.3484-16.2017

    • PubMed
    • Google Scholar
69. 1. Wagenmakers E-J
    2. Van Der Maas HLJ
    3. Grasman RPPP

    (2007) An Ez-diffusion model for response time and accuracy

    Psychonomic Bulletin & Review 14:3–22.

    https://doi.org/10.3758/BF03194023

    • Google Scholar
70. 1. Wandell BA
    2. Winawer J

    (2011) Imaging Retinotopic maps in the human brain

    Vision Research 51:718–737.

    https://doi.org/10.1016/j.visres.2010.08.004

    • PubMed
    • Google Scholar
71. 1. Womelsdorf T
    2. Anton-Erxleben K
    3. Pieper F
    4. Treue S

    (2006) Dynamic shifts of visual receptive fields in cortical area mt by spatial attention

    Nature Neuroscience 9:1156–1160.

    https://doi.org/10.1038/nn1748

    • PubMed
    • Google Scholar
72. 1. Yamins DLK
    2. Hong H
    3. Cadieu CF
    4. Solomon EA
    5. Seibert D
    6. DiCarlo JJ

    (2014) Performance-Optimized Hierarchical models predict neural responses in higher visual cortex

    PNAS 111:8619–8624.

    https://doi.org/10.1073/pnas.1403112111

    • Google Scholar
73. 1. Zirnsak M
    2. Steinmetz NA
    3. Noudoost B
    4. Xu KZ
    5. Moore T

    (2014) Visual space is compressed in Prefrontal cortex before eye movements

    Nature 507:504–507.

    https://doi.org/10.1038/nature13149

    • PubMed
    • Google Scholar

Decision letter after peer review:

Thank you for submitting your article "Behavioral benefits of spatial attention explained by multiplicative gain, not receptive field shifts, in a neural network model" for consideration by eLife. Your article has been reviewed by 2 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Floris de Lange as the Senior Editor. The following individuals involved in the review of your submission have agreed to reveal their identity: Albert Compte (Reviewer #1); Kendrick N Kay (Reviewer #2).

The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

We were all very enthusiastic about the approach and the findings. During our discussions, we collectively focused on several issues regarding the generality of the conclusions and the methods used to arrive at them. Below is a list of the main issues:

1) The Detection vs discrimination. Having at least one other task would allow you to determine if the gain story is specific to detection or if the type of modulation depends on the task. In either case, results from another task would be interesting and would greatly improve the generality and theoretical impact of the claims.

2) Justifying the spatial invariance of the readout mechanism and potentially testing other schemes.

3) What happens when gain isn't just injected into "V1", but into other layers? For example, Lindsay has shown that this matters in the context of CNNs performing attention tasks, and the rationale for injecting a change so early in the system isn't entirely clear.

4) Others, like Compte and Wang 2006, have also explored this question using other architectures. Given that the CORnet-z model does not include recurrent dynamics within each layer, it will be important to discuss and consider the implications of the chosen network and implications for the generality of the conclusions.

Reviewer #1 (Recommendations for the authors):

1) The model requires a more detailed presentation in the text (now lines 93,94). A succinct description of how it is built should be provided so the reader does not have to look up the references for a simple, broad-picture understanding. It is important to mention that the model has been built and trained previously based on the same set of images that will be used here, and the weights are available, so the only further training done here is the linear classifier on the model outputs to detect image categories upon presentation of "composite" grids. It took a while for me to understand this and to see the difference between base images and composite images. There is also a confusing thing about the neural model in Figure 2d: it pictures the ReLU occurring after the max pooling step, but in Figure 1 of Kubilius et al. (2018) it occurs before the max pooling step. The sentence ("Unit activations were measured after the convolution, prior to the max pooling step") in the caption of Figure 2 is also confusing: how is unit activity being measured before the ReLU non-linearity that transforms inputs into output rates?

2) The manuscript emphasizes 3 different RF modulations: scaling, shift, and shrinkage, but then only two of them are specifically isolated with network simulations. RF shrinkage is not addressed. This is not really a problem, but it would be nice to have full "symmetry" in the manuscript. Is there a specific reason why this modulation can not be assessed to make it more parallel to the rest?

3) References are required in lines 145 ("…behavioral benefits of attention") and 250 ("…as others have suggested").

4) In general, scatter plots in the figures should indicate what are the individual data points presented. I think in some cases are units, in other cases they are categories. This should be clearly indicated.

Reviewer #2 (Recommendations for the authors):1. On the issue of the readout.As mentioned in the public review, it appears that the conclusions are heavily dependent on the choice of readout mechanism. Thus, it seems a bit premature at this point to make strong general conclusions about which tuning properties are critical for behavioral improvements. Many recent studies have shown spatial tuning (i.e. large but limited RF sizes) in the high-level visual cortex; hence, in the neural network observer model, it is not clear why the readout (linear classification) is performed on unit activity that has been summed fully across visual space.

Now, it is an open interesting empirical question whether perceptual decisions from the visual system are based on fully spatially invariant units. But note that if that were the case, isn't one counterintuitive implication that our perceptual decisions can be corrupted by visual stimuli far away from the relevant decision zone?

If we entertain the possibility that the readout can be from units that have spatial specificity, then it remains to be seen what sort of modeling outcomes there might be.

Even though the authors show, in the context of their particular model, that tuning shifts do not produce the required behavioral improvements, it seems that in general, it should be possible in theory for shifts to provide the required benefits. For example, if the focal cue is on stimuli in the upper right quadrant, if all receptive fields in the model covering the other quadrants were fully shifted to the upper right quadrant, wouldn't the readout from the model obviously improve?

With regards to the linear classifiers implemented by the authors, it seems there are many important details that need to get spelled out. Things like how exactly was the final predicted category obtained? Was a 20-way classification scheme used, or some winner-take-all scheme from 20 separate binary classifiers used? Was there any regularization used to fit the weights of the logistic regression? If not, how can we be confident that the amount of training data was sufficient? Details on the classification methods are important for interpreting how upstream computational changes impact the behavioral output from the model.

2. Reframing and/or exploring other models.

In light of the uncertainties about how general the conclusions can be from the modeling results, the authors may wish to consider refocusing/reframing their work. Instead of attempting to make strong general conclusions about the relationship between behavior and computational mechanisms, the author might reframe their work into more of a computational and theory-building exercise.

In particular, the authors show interesting insights regarding how a Gaussian applied to an early layer of a CNN can have effects that propagate through the network and which manifest as complex, rich RF tuning changes elsewhere in the model (e.g. as the authors state, "RF shifts are a result of the signal gain…") (see also Compte Cerebral Cortex 2006). The distinction between architecture (internal computational mechanisms) and what might manifest at the level of input-output behavior of a unit is an important one, as the authors discuss (p. 17, lines 257-268). Certainly, a number of previous studies have adopted the latter mindset. Note that doing so is not necessarily inconsistent with the former "mechanistic" mindset (they are just different modeling stances one can take). Emphasizing and generating insights from adopting a "mechanistic" mindset is valuable research output.

Thus, one suggestion is to reframe the work to focus more on this insight and think about the theoretical ramifications (and/or concrete experimental predictions) that might stem from this. Related to this effort might be to consider other models. For example, it would be interesting to understand the similarities and differences between the 'attentional field' model (Reynolds, Heeger) and the current model. As another example, it would be interesting to consider models in which the "Gaussian gain" is applied at the end (deep layers) and effects propagate backwards, as opposed to being applied at the beginning and effects propagating forwards.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Thank you for resubmitting your work entitled "Gain, not concomitant changes in spatial receptive field properties, improves task performance in a neural network attention model" for further consideration by eLife.

Reviewer #2 and I have re-evaluated your revised submission and we both agree that it is much improved and makes a strong contribution to the literature. There are a few remaining issues that need to be addressed that are captured in Reviewer #2's comments, and I'd ask you to address them before a final decision is made. Please let me know if you have any questions, and thanks.

Reviewer #2 (Recommendations for the authors):

I have reviewed the revisions and the authors' responses.

Overall, the revisions are solid and substantial. The additions to the paper are useful, in particular, the new analyses exploring exactly how spatial information is included, restricted, or "averaged over", which provide insights into the potential equivalency of the gain effects and the selection (or suppression) of irrelevant information. In addition, the new framing and discussion provide helpful clarity over the original manuscript and make for a more straightforward read. The computational modeling performed in this paper represents a novel and well-executed foray into a challenging area. While it does not necessarily definitively resolve the issues at stake, I believe it provides and demonstrates an interesting and non-trivial approach.

A few critical comments on the revisions:

The authors state, "At larger scales or in other tasks there are theoretical reasons to expect that task performance will improve due to these effects". In this sentence, the referent of "these effects" is a bit unclear. Also, I am not sure what 'larger scales' refers to. Finally, it would be useful to give specific examples of what types of 'other tasks' might be implied by the sentence here.

In response to the reviews, the authors added some detail on the linear classifier methods, but the text is still not sufficient. More detail on exactly what was implemented seems important. For example, details on the "under the curve" quantification. In addition, perhaps there is a misspecification of the methods: As the authors state, "Weights were fit using logistic regression with an L2 loss and no regularization, using scikit-learn and the LIBLINEAR package (Pedregosa et al., 2011)." However, if I understand correctly, for each category, there are effectively 512 parameters that need to get learned and only 100 instances to train these parameters. Hence, if no regularization were used, this presumably would lead to a very noisy (overfit) solution. Perhaps the authors meant that they used L2 regularization (and the default settings for the regularization hyperparameter in the scikit-learn package)? Note that logistic regression implies a probabilistic formulation of the classification problem, and would seem to suggest that the authors don't actually mean that they used an L2 loss (which implies additive Gaussian noise). Hopefully, these important analysis choices and details can be clarified and do not have larger ramifications for the rest of the paper.

Essential revisions:

We were all very enthusiastic about the approach and the findings. During our discussions, we collectively focused on several issues regarding the generality of the conclusions and the methods used to arrive at them. Below is a list of the main issues:

1) The Detection vs discrimination. Having at least one other task would allow you to determine if the gain story is specific to detection or if the type of modulation depends on the task. In either case, results from another task would be interesting and would greatly improve the generality and theoretical impact of the claims.

We agree that generalization to a discrimination task is worth testing. To ensure our results generalize better we ran our analysis against such a task using an analog to a two-interval forced choice task and found qualitatively similar results to the detection task [Figure 8 Supplement 2].

2) Justifying the spatial invariance of the readout mechanism and potentially testing other schemes.

Agreed that the assumption of spatial invariance in the original model is important to test. To ensure that our results were not affected by the choice to read out from a spatially collapsed final layer, we now report two new analyses: first, the result for a model trained to read out from the full spatial output (7 x 7 x 512) [292-308] and second, the result for a model trained to read out only from the target quadrant while masking out the other four. The results from these analyses support our initial conclusions, please see the text for the rationale and results [Figure 8 Supplement 1].

3) What happens when gain isn't just injected into "V1", but into other layers? For example, Lindsay has shown that this matters in the context of CNNs performing attention tasks, and the rationale for injecting a change so early in the system isn't entirely clear.

We’ve clarified in the text that the propagated gain test which is applied at the last layer of the CNN (Figure 8) demonstrates that for this particular model the choice of layer only changes whether the receptive field properties are changed, not the task performance [274-283]. We’ve also added a more complete discussion of the rationale for injecting gain into the earlier layers [442-448] and the Lindsay paper specifically [453-456].

4) Others, like Compte and Wang 2006, have also explored this question using other architectures. Given that the CORnet-z model does not include recurrent dynamics within each layer, it will be important to discuss and consider the implications of the chosen network and implications for the generality of the conclusions.

We’ve added a more complete discussion of this issue in the discussion [411-424].

Reviewer #1 (Recommendations for the authors):

1) The model requires a more detailed presentation in the text (now lines 93,94). A succinct description of how it is built should be provided so the reader does not have to look up the references for a simple, broad-picture understanding. It is important to mention that the model has been built and trained previously based on the same set of images that will be used here, and the weights are available, so the only further training done here is the linear classifier on the model outputs to detect image categories upon presentation of "composite" grids. It took a while for me to understand this and to see the difference between base images and composite images. There is also a confusing thing about the neural model in Figure 2d: it pictures the ReLU occurring after the max pooling step, but in Figure 1 of Kubilius et al. (2018) it occurs before the max pooling step. The sentence ("Unit activations were measured after the convolution, prior to the max pooling step") in the caption of Figure 2 is also confusing: how is unit activity being measured before the ReLU non-linearity that transforms inputs into output rates?

We’ve added a better description of the model in both the methods [564-590] and results [109-113]. We thank the reviewer for pointing out the error in the figure and text concerning the order of the relu and maxpool operations and have fixed these.

2) The manuscript emphasizes 3 different RF modulations: scaling, shift, and shrinkage, but then only two of them are specifically isolated with network simulations. RF shrinkage is not addressed. This is not really a problem, but it would be nice to have full "symmetry" in the manuscript. Is there a specific reason why this modulation can not be assessed to make it more parallel to the rest?

We very much appreciate the suggestion to make the paper symmetrical and have added a new figure and series of analyses showing the effects of shrinkage [Figure 6, 222-245].

3) References are required in lines 145 ("…behavioral benefits of attention") and 250 ("…as others have suggested").

Citations have been added for each statement.

4) In general, scatter plots in the figures should indicate what are the individual data points presented. I think in some cases are units, in other cases they are categories. This should be clearly indicated.

We’ve clarified whether data points indicate units or categories for all figures.

Reviewer #2 (Recommendations for the authors):

1. On the issue of the readout.

As mentioned in the public review, it appears that the conclusions are heavily dependent on the choice of readout mechanism. Thus, it seems a bit premature at this point to make strong general conclusions about which tuning properties are critical for behavioral improvements. Many recent studies have shown spatial tuning (i.e. large but limited RF sizes) in the high-level visual cortex; hence, in the neural network observer model, it is not clear why the readout (linear classification) is performed on unit activity that has been summed fully across visual space.

We agree that it is important to consider models with some spatial tuning at the decision stage to more accurately model ventral temporal cortical inputs that are expected to have such tuning. We have added a model variant to address this issue and have arrived at similar conclusions, see [292-301] for details.

Now, it is an open interesting empirical question whether perceptual decisions from the visual system are based on fully spatially invariant units. But note that if that were the case, isn't one counterintuitive implication that our perceptual decisions can be corrupted by visual stimuli far away from the relevant decision zone?

Yes, we agree. We would clarify though that we are considering both visual processing and the decision stage [see 424-440]. At the decision stage, uncertainty over which visual signal to base a decision on is a problem that attention has been proposed to solve (e.g. Pelli, 1985). In this context, set size effects in which increasing the size of stimulus arrays across the visual field increases response times (even for covert search), can be viewed as the effect of uncertainty introduced by visual stimuli far from a search target. Or, for example in our own work, in which cueing one of four spatially separate targets improves contrast discrimination thresholds compared to distributing attention (Pestilli et al., 2011), even though distractors are far from the post-cued decision target.

If we entertain the possibility that the readout can be from units that have spatial specificity, then it remains to be seen what sort of modeling outcomes there might be.

Please see [309-319] where we show that the modeling outcomes for considering spatially specific responses does not qualitatively change our conclusions.

Even though the authors show, in the context of their particular model, that tuning shifts do not produce the required behavioral improvements, it seems that in general, it should be possible in theory for shifts to provide the required benefits. For example, if the focal cue is on stimuli in the upper right quadrant, if all receptive fields in the model covering the other quadrants were fully shifted to the upper right quadrant, wouldn't the readout from the model obviously improve?

We do agree that, in principle, tuning shifts could provide required benefits. We would like to clarify that we are considering the tuning shifts that occur because of downstream effects of increasing gain of a magnitude required to match human performance benefits with attention cueing. In our reframing of the manuscript (see below), we have substantially rewritten the text to clarify this particular point, and that our conclusion is *not* that tuning shifts of any magnitude won’t have behavioral effects [394-395 and 441-467], but is specific to those that occur with gain changes. We have also specifically tested the case pointed out above in which only receptive fields covering the target are considered and receptive field shifts of the magnitude found for the gain change are considered, We find that the magnitude of these changes is not enough to induce task performance benefits for the observer model. See [320-328].

With regards to the linear classifiers implemented by the authors, it seems there are many important details that need to get spelled out. Things like how exactly was the final predicted category obtained? Was a 20-way classification scheme used, or some winner-take-all scheme from 20 separate binary classifiers used? Was there any regularization used to fit the weights of the logistic regression? If not, how can we be confident that the amount of training data was sufficient? Details on the classification methods are important for interpreting how upstream computational changes impact the behavioral output from the model.

These details have now been added in the results [117-119] and methods [570-590]. We believe that the training data was sufficient because the validation set showed stable classification accuracy [118-119].

2. Reframing and/or exploring other models.

In light of the uncertainties about how general the conclusions can be from the modeling results, the authors may wish to consider refocusing/reframing their work. Instead of attempting to make strong general conclusions about the relationship between behavior and computational mechanisms, the author might reframe their work into more of a computational and theory-building exercise.

In particular, the authors show interesting insights regarding how a Gaussian applied to an early layer of a CNN can have effects that propagate through the network and which manifest as complex, rich RF tuning changes elsewhere in the model (e.g. as the authors state, "RF shifts are a result of the signal gain…") (see also Compte Cerebral Cortex 2006). The distinction between architecture (internal computational mechanisms) and what might manifest at the level of input-output behavior of a unit is an important one, as the authors discuss (p. 17, lines 257-268). Certainly, a number of previous studies have adopted the latter mindset. Note that doing so is not necessarily inconsistent with the former "mechanistic" mindset (they are just different modeling stances one can take). Emphasizing and generating insights from adopting a "mechanistic" mindset is valuable research output.

Thus, one suggestion is to reframe the work to focus more on this insight and think about the theoretical ramifications (and/or concrete experimental predictions) that might stem from this. Related to this effort might be to consider other models. For example, it would be interesting to understand the similarities and differences between the 'attentional field' model (Reynolds, Heeger) and the current model. As another example, it would be interesting to consider models in which the "Gaussian gain" is applied at the end (deep layers) and effects propagate backwards, as opposed to being applied at the beginning and effects propagating forwards.

Thank you for this helpful suggestion. We have substantially modified the manuscript throughout to reframe as suggested. In particular, please note the new title, abstract, changes in the introduction [36-38, 49-52, 59-64] and re-written discussion. We also have extended our discussion of the alternate model (what we call propagated gain model) of adding gain at the last stages (end, deep layers) of the CNN [274-328]. The RH models consider the effects of gain in a similar fashion but only on a single layer model, which we now mention in the discussion [435-437]

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Reviewer #2 (Recommendations for the authors):

I have reviewed the revisions and the authors' responses.

Overall, the revisions are solid and substantial. The additions to the paper are useful, in particular, the new analyses exploring exactly how spatial information is included, restricted, or "averaged over", which provide insights into the potential equivalency of the gain effects and the selection (or suppression) of irrelevant information. In addition, the new framing and discussion provide helpful clarity over the original manuscript and make for a more straightforward read. The computational modeling performed in this paper represents a novel and well-executed foray into a challenging area. While it does not necessarily definitively resolve the issues at stake, I believe it provides and demonstrates an interesting and non-trivial approach.

A few critical comments on the revisions:

The authors state, "At larger scales or in other tasks there are theoretical reasons to expect that task performance will improve due to these effects". In this sentence, the referent of "these effects" is a bit unclear. Also, I am not sure what 'larger scales' refers to. Finally, it would be useful to give specific examples of what types of 'other tasks' might be implied by the sentence here.

We agree that these additional details help clarify this important discussion point. We have clarified the meaning of each term and specifically mention judgments of spatial position as an example of a task that should theoretically show a larger benefit from shift/shrinkage of receptive fields. [new text at lines 460-472].

In response to the reviews, the authors added some detail on the linear classifier methods, but the text is still not sufficient. More detail on exactly what was implemented seems important. For example, details on the "under the curve" quantification.

We have added additional detail to this section, explaining how the AUC is calculated and how it can be interpreted [lines 681-690].

In addition, perhaps there is a misspecification of the methods: As the authors state, "Weights were fit using logistic regression with an L2 loss and no regularization, using scikit-learn and the LIBLINEAR package (Pedregosa et al., 2011)." However, if I understand correctly, for each category, there are effectively 512 parameters that need to get learned and only 100 instances to train these parameters. Hence, if no regularization were used, this presumably would lead to a very noisy (overfit) solution. Perhaps the authors meant that they used L2 regularization (and the default settings for the regularization hyperparameter in the scikit-learn package)? Note that logistic regression implies a probabilistic formulation of the classification problem, and would seem to suggest that the authors don't actually mean that they used an L2 loss (which implies additive Gaussian noise). Hopefully, these important analysis choices and details can be clarified and do not have larger ramifications for the rest of the paper.

We thank the reviewer for raising these important concerns, which were partly due to a typo. We have fixed the typo to clarify that L2 regularization was used in training the output layers, specifically to reduce the risk of overfitting to the training data [Methods lines 673-680]. We have also made it more clear that the output layers were evaluated on a held-out validation set (median AUC 0.9), providing further evidence that the models are not overfit [Results lines 194-195] and we have copied this information into the methods to make it easier to find [Methods lines 679-680]

Author details

1. Kai J Fox

   Department of Psychology, Stanford University, Stanford, United States

   Contribution

   Conceptualization, Software, Formal analysis, Investigation, Visualization, Methodology, Writing – review and editing

   Contributed equally with

   Daniel Birman

   For correspondence

   kaifox@stanford.edu

   Competing interests

   No competing interests declared

2. Daniel Birman

   Department of Psychology, Stanford University, Stanford, United States

   Present address

   Department of Biological Structure, University of Washington, Washington, United States

   Contribution

   Conceptualization, Formal analysis, Supervision, Investigation, Writing – original draft, Writing – review and editing

   Contributed equally with

   Kai J Fox

   For correspondence

   dbirman@uw.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-3748-6289

3. Justin L Gardner

   Department of Psychology, Stanford University, Stanford, United States

   Contribution

   Conceptualization, Supervision, Funding acquisition, Project administration, Writing – review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-2220-5050

• 351

  Page views

• 80

  Downloads

• 1

  Citations

Article citation count generated by polling the highest count across the following sources: PubMed Central, Crossref, Scopus.