Our senses are constantly bombarded by sights and sounds, but the capacity of the brain to process all these inputs is finite. The stimuli that contain the most useful information must therefore be prioritized for processing by the brain to ensure that we build up as complete a picture as possible of the world around us. However, the strategy that the brain uses to select certain stimuli—or certain features of stimuli—for processing at the expense of others is unclear.

Hermundstad et al. have now provided new insights into this process by analyzing how humans respond to artificial stimuli that contain controllable mixtures of features that found in natural stimuli. To do this, Hermundstad et al. selected photographs of the natural world, and measured the brightness of individual pixels. After adjusting images in a way that mimics the human retina, the brightest 50% of the pixels in each photograph were colored white and the remaining 50% were colored black.

Hermundstad et al. then used statistical techniques to calculate the degree to which the color of pixels could be used to predict the color of their neighbors. In this way, it was possible to calculate the amount of variation throughout the images, and then make computer-generated images in which pixel colorings were more or less predictable than in the natural images.

Volunteers then performed a task in which they had to locate a computer-generated pattern against a background of random noise. The volunteers were able to locate this target most easily when it contained the same kinds of patterns and features that were meaningful about natural images.

While this shows that the brain is adapted to prioritize features that are more informative about the natural world, understanding exactly how the brain implements this strategy remains a challenge.

Sensory receptor neurons encode signals from the environment, which are then transformed by successive neural layers to support diverse and computationally complex cognitive tasks. A normative understanding of these computations begins in the periphery, where the efficient coding principle—the notion that a sensory system is tuned to the statistics of its natural inputs—has been shown to be a powerful organizing framework (Barlow, 2001; Simoncelli, 2002). Perhaps the best-known example is that of redundancy removal via predictive coding and spatiotemporal decorrelation. In insects, this is carried out by neural processing (Laughlin, 1981; van Hateren, 1992b); in vertebrates, fixational eye movements—which precede the first step of neural processing (Srinivasan et al., 1982; Atick and Redlich, 1990; Atick et al., 1992)—play a major role (Kuang et al., 2012). This approach was later extended to describe population coding, retinal mosaic structure (Barlow, 2001; Karklin and Simoncelli, 2001; Borghuis et al., 2008; Balasubramanian and Sterling, 2009; Liu et al., 2009; Garrigan et al., 2010; Ratliff et al., 2010; Kuang et al., 2012), adaptation of neural responses (Brenner et al., 2000; Fairhall et al., 2001; Schwartz and Simoncelli, 2001), and early auditory processing (Smith and Lewicki, 2006). Taken together, normative theories based on efficient coding have been successful in explaining aspects of processing in the sensory periphery that are tuned to simple statistical features of the natural world.

Can we extend such theories beyond the sensory periphery to describe cortical sensitivity to complex sensory features? Normative theories have been successful in predicting the response properties of single cells, including receptive fields in V1 (Olshausen and Field, 1996; Bell and Sejnowski, 1997; van Hateren and Ruderman, 1998; van Hateren and van der Schaaf, 1998; Hyvarinen and Hoyer, 2000; Vinje and Gallant, 2000; Karklin and Lewicki, 2009) and spectro-temporal receptive fields in primary auditory cortex (Carlson and DeWeese, 2002, 2012), as well as distributions of tuning curves across individual cells in a population (Lewicki, 2002; Ganguli and Simoncelli, 2011). Some complex features, however, might not be represented by the tuning properties of individual cells in any direct way, but rather emerge from the collective behavior of many cells. Instead of trying to predict individual cell properties, we therefore focus on the sensitivity of the complete neural population. Is there an organizing principle that determines how resources within the population are allocated to representing such complex features?

When the presence of complex features is predictable (i.e., can be accurately guessed from simpler features along with priors about the environment), mechanisms are best devoted elsewhere (See Discussion, van Hateren, 1992a). In contrast, sensory features that are highly variable and not predictable from simpler ones can serve to determine their causes (e.g., to distinguish among materials or objects), a first step in guiding decisions. We will show that these ideas predict a specific organizing principle for aggregate sensitivities arising in cortex: the perceptual salience of complex sensory signals increases with the variability, or unpredictability, of the corresponding signals over the ensemble of natural stimuli.

To test this hypothesis, we focus on early stages of central visual processing. Here, early visual cortex (V1 and V2) is charged with extracting edges, shapes, and other complex correlations of light between multiple points in space (Morrone and Burr, 1988; Oppenheim and Lim, 1981; von der Heydt et al., 1984). We compare the spatial variation of local patterns of light across natural images with human sensitivity to manipulations of the same patterns in synthetic images. This allows us to determine how sensitivity is distributed across many different features, rather than simply determining the most salient ones. (We will say that a feature is more salient if it is more easily discriminated from white noise.) To this end, we parametrize the space of local multi-point correlations in images in terms of a complete set of coordinates, and we measure the probability distribution of coordinate values sampled over a large ensemble of natural scenes. We then use a psychophysical discrimination task to measure human sensitivity to the same correlations in synthetic images, where the correlations can be isolated and manipulated in a mathematically rigorous fashion by varying the corresponding coordinates (Chubb et al., 2004; Victor et al., 2005; Victor and Conte, 2012; Victor et al., 2013). Comparing the measurements, we show that human sensitivity to these multi-point elements of visual form is tuned to their variation in the natural world. Our result supports a broad hypothesis: cortex invests preferentially in mechanisms that encode unpredictable sensory features that are more variable, and thus more informative about the world. Namely, variance is salience.

As we recently showed, some informative local correlations of natural scenes are captured by the configurations of luminances seen through a ‘glider’, that is, a window defined by a 2 × 2 square arrangement of pixels (Tkačik et al., 2010). We use this observation first as a framework for analyzing the local statistical structure of natural scenes, then to characterize psychophysical sensitivities via a set of synthetic visual texture stimuli, and finally to compare the two.

Analyzing local image statistics in natural scenes

The analysis of natural scenes is schematized in Figure 1. We collect an ensemble of image patches from the calibrated Penn natural image database (PIDB) (Tkačik et al., 2011). We preprocess the image patches as shown in Figure 1A. This involves first averaging pixel luminances over a square region of N × N pixels, which converts an image of size L₁ × L₂ pixels into an image of reduced size L₁/N × L₂/N pixels. Images are then divided into R × R square patches of these downsampled pixels and whitened (see ‘Materials and methods’, Image preprocessing, for further details). Since the preprocessing depends on a choice of two parameters, the block-average factor N and patch size R, we report results for multiple image analyses performed using the identical preprocessing pipeline but for various choices of N and R. After preprocessing, we binarize each patch to have equal numbers of black and white pixels (black = −1, white = +1). We characterize each patch by the histogram of 16 binary colorings (2^2×2) seen through a square 2 × 2 pixel glider (Figure 1B). Translation invariance imposes constraints on this histogram, reducing the number of degrees of freedom to 10 (Victor and Conte, 2012). These degrees of freedom can be mapped to a set of image statistic coordinates that separates correlations based on their order: (i) one first-order coordinate, γ, describes overall luminance, (ii) four second-order coordinates, $\{{\beta }_{|}$, ${\beta }_{-}$, ${\beta }_{/}$, ${\beta }_{\backslash }\}$, describe two-point correlations between pixels arranged vertically, horizontally, or diagonally, (iii) four third-order coordinates, $\{{\theta }_{\mathrm{⌞}}$, ${\theta }_{\mathrm{⌜}}$, ${\theta }_{\mathrm{⌝}}$, ${\theta }_{\mathrm{⌟}}\}$, describe three-point correlations between pixels arranged into ⌞-shapes of different orientations, and (iv) one fourth-order coordinate, α, describes the single four-point correlation between all four pixels in the glider (Figure 1C). The binarization step of the preprocessing pipeline forces γ to zero, leaving nine coordinates. Each image patch is thus characterized by a vector of coordinate values $\left\{{\beta }_{|},{\beta }_{-},{\beta }_{/},{\beta }_{\backslash },{\theta }_{\mathrm{⌜}},{\theta }_{\mathrm{⌝}},{\theta }_{\mathrm{⌟}},{\theta }_{\mathrm{⌞}},\alpha \right\}$, that is, a point within the multidimensional space of image statistics. Accumulating these points across patches yields a multidimensional probability distribution that characterizes the local correlations in natural scenes (schematized in Figure 1D). A total of 724 images (up to 249780 patches, depending on the choice of N and R), was used to construct this distribution.

Figure 1 with 3 supplements see all

Download asset Open asset

To summarize this distribution, we compute the degree of variation (standard deviation) along each coordinate axis (Figure 1E). As is shown, the degree of variation along different coordinate axes exhibits a characteristic rank-ordering, given by $\left\{{\beta }_{|},{\beta }_{-}\right\}>\left\{{\beta }_{/},{\beta }_{\backslash }\right\}>\alpha >\left\{{\theta }_{\mathrm{⌞}},{\theta }_{\mathrm{⌜}},{\theta }_{\mathrm{⌝}},{\theta }_{\mathrm{⌟}}\right\}$; that is, the most variable correlations are pairwise correlations in the cardinal directions, followed by pairwise correlations in the oblique directions, followed by fourth-order correlations. Interestingly, third-order correlations are the least variable across image patches. An analogous analysis performed on white noise yields a flat distribution with considerably smaller standard deviation values (See ‘Materials and methods’, Analysis variants for Penn Natural Image Database, and Figure 1—figure supplement 3 for comparison), and performing the analysis on a colored Gaussian noise (e.g. $1/f^k$ spectrum) would also yield a flat distribution because of the whitening stage in the image preprocessing pipeline. These (and subsequent) findings are preserved across different choices of image analysis parameters (shown in Figure 1E for block-average factors N = 2, 4 and patch sizes R = 32, 48, 64; see ‘Materials and methods’, Analysis variants for Penn Natural Image Database, and Figure 3—figure supplement 5A for a larger set of parameters) and also across other collections of natural images (see ‘Materials and methods’, Comparison with van Hateren Database, and Figure 3—figure supplement 5B for a parallel analysis of the van Hateren image dataset (van Hateren and van der Schaaf, 1998), which gives similar results).

Characterizing visual sensitivity to local image statistics

To characterize perceptual sensitivity to different statistics, we isolated them in synthetic visual images and used a figure/ground segmentation task (Figure 2B). We used a four-alternative forced-choice task in which stimuli consisted of a textured target and a binary noise background (or vice-versa). Each stimulus was presented for 120ms and was followed by a noise mask. Subjects were then asked to identify the spatial location (top, bottom, left, or right) of the target. Experiments were carried out for synthetic stimuli in which the target or background was defined by first varying image statistic coordinates independently (Figure 2A shows examples of gamuts from which stimuli are built). Along each coordinate axis, threshold (1/sensitivity) was defined as the coordinate value required to support a criterion level of performance (Figure 2C, inset). We then performed further experiments in which the target or background was defined by simultaneously varying pairs of coordinates. For measurements involving each coordinate pair (to which we will refer as a ‘coordinate plane’), we traced out an isodiscrimination contour (Figure 2C) that describes the threshold values not only along the cardinal directions, but also along oblique directions. Measurements were collected for four individual subjects in each of 11 distinct coordinate planes (representing all distinct coordinate pairs up to 4-fold rotational symmetry; see ‘Materials and methods’, Psychophysical methods, for further details). Each subject performed 4320 judgements per plane, for a total of 47,520 trials per subject.

Figure 2

Download asset Open asset

Figure 2D shows perceptual sensitivities measured along each coordinate axis. For each of four subjects, a similar pattern emerges for sensitivities as was observed for variation in natural image statistics: sensitivities are rank-ordered as $\left\{{\beta }_{|},{\beta }_{-}\right\}>\left\{{\beta }_{/},{\beta }_{\backslash }\right\}>\alpha >\left\{{\theta }_{\mathrm{⌜}},{\theta }_{\mathrm{⌝}},{\theta }_{\mathrm{⌟}},{\theta }_{\mathrm{⌞}}\right\}$.

Note that the difference between the sensitivities in the horizontal and vertical directions (${\beta }_{-}$ and ${\beta }_{|}$) vs the diagonal directions (${\beta }_{\backslash }$ and ${\beta }_{/}$) is not simply an ‘oblique effect’, that is, a greater sensitivity to cardinally- vs obliquely-oriented contours (Campbell et al., 1966). Horizontal and vertical pairwise correlations differ from the diagonal pairwise correlations in more than just orientation: pixels involved in horizontal and vertical pairwise correlations share an edge, while pixels involved in diagonal pairwise correlations only share a corner. Correspondingly, the difference in sensitivities for horizontal and vertical correlations vs diagonal correlations is approximately 50%, which is much larger than the size of the classical oblique effect (10–20%) (Campbell et al., 1966).

Natural scenes predict human sensitivity along single coordinates

Figures 1E and 2D show a rank-order correspondence between natural image statistics and perceptual sensitivities. This qualitative comparison can be converted to a quantitative one (Figure 3A), as a single scaling parameter aligns the standard deviation of natural image statistics with the corresponding perceptual sensitivities. In this procedure, each of the six image analyses is scaled by a single multiplicative factor that minimizes the squared error between the set of standard deviations and the set of subject-averaged sensitivities (see ‘Materials and methods’, Image preprocessing, and Figure 3—figure supplement 1 for additional details regarding scaling). The agreement is very good, with the mismatch between image analyses and human psychophysics comparable to the variability from one image analysis to another, or from one human subject to another.

Figure 3 with 9 supplements see all

Download asset Open asset

We quantify the correspondence between image analyses and psychophysical analyses by computing the scalar product between the normalized vector of standard deviations (extracted separately from each image analysis) and the normalized vector of subject-averaged sensitivities (extracted from the set of psychophysical analyses). A value of 1 indicates perfect correspondence, and 0 indicates no correspondence. This value ranges from 0.987 to 0.999 across image analyses and is consistently larger than the value measured under the null hypothesis that the apparent correspondence between statistics and sensitivities is chance (p ≤ .0003 for each image analysis; see Tables 1–2 and ‘Materials and methods’, Permutation tests, for details regarding statistical tests).

These findings support our hypothesis that human perceptual sensitivity measured along single coordinate axes (assessed using synthetic binary textures) is predicted by the degree of variation along the same coordinate axes in natural scenes.

How should neural mechanisms be distributed to represent a diverse set of informative sensory features? We argued that, when performance requires inferences limited by sampling of the statistics of input features, resources should be devoted in proportion to feature variability. A basic idea here is that features that take a wider range of possible values are less predictable, and will better distinguish between contexts in the face of input noise. We used this hypothesis to successfully predict human sensitivity to elements of visual form arising from spatial multi-point correlations in images. This result is notable for several reasons. First, we successfully predicted dozens of independent parameters that describe human perceptual sensitivity. The only free parameter was a scale that converted between perceptual sensitivities and natural image statistics. Moreover, predictions about the rank ordering of sensitivities (Figure 3A) and the shape and orientation of isodiscrimination contours (Figure 3B) do not even require a scale factor. Second, the theoretical predictions and their psychophysical test were derived from two very different sources. Psychophysical stimuli consisted of mathematically-defined synthetic binary textures with precisely-controlled correlational structure that is unlikely to occur outside of the laboratory. In contrast, the efficient coding predictions were derived from calibrated photographs of natural scenes in which many types of correlations are simultaneously present. Third, predictions refer to multi-point (and not just pairwise) correlations, which are critical for defining local features such as lines and edges (Oppenheim and Lim, 1981; Morrone and Burr, 1988). In contrast, previous normative theories have have mainly focused on explaining the linear receptive fields of neurons in primary visual (Olshausen and Field, 1996; Bell and Sejnowski, 1997; van Hateren and Ruderman, 1998; van Hateren and van der Schaaf, 1998; Hyvarinen and Hoyer, 2000; Vinje and Gallant, 2000; Karklin and Lewicki, 2009) and auditory cortex (Carlson and DeWeese, 2002, 2012), or on deriving symmetry- and coverage-based mesoscopic models of cortical map formation in V1 (Wolf and Geisel, 1998; Swindale et al., 2000; Kaschube et al., 2011). Finally, the efficient coding prediction of greater sensitivity to more variable multipoint correlations is closely tied to the statistical structure of natural visual images. Specifically, this regime applies to highly variable multipoint correlations that cannot be predicted from simpler ones. Some other multipoint correlations (defined on configurations other than a 2 × 2 glider) are also highly variable, but they are predictable from simpler correlations. For these multipoint correlations, visual sensitivity is very low (Tkačik et al., 2010), and efficient coding is not applicable in the form proposed here.

In sum, the surprising predictive power and the high statistical significance of our results provide strong support for the proposed application of the efficient coding hypothesis to cortical processing of complex sensory features.

Cortex faces a different class of challenges than the sensory periphery

Successive stages of sensory processing share the same broad goals: invest resources in encoding stimulus features that are sufficiently informative, and suppress less-informative ones. In the periphery, this is exemplified by the well-known suppression of very low spatial frequencies; in cortex, this is exemplified by insensitivity to high-order correlations that are predictable from lower-order ones. Previous work has shown that such higher-order correlations can be separated into two groups—informative and uninformative—and only the informative ones are encoded (Tkačik et al., 2010). We used this finding to select an informative subspace for the present study, and we asked how resources should be efficiently allocated amongst features within this informative subspace.

A simple model of efficient coding by neural populations is shown in Figure 4A (details in ‘Materials ans methods’, Two regimes of efficient coding). Here, to enable analytical calculations, we used linear filters of variable gain and subject to Gaussian noise to model a population of neural channels encoding different features. The optimal allocation of resources to maximize information transmitted by the population depends on the amount of input noise, the amount of output noise, the input signal variability, and the total resources available to the system, here quantified as a constraint on the total output power (i.e., sum of response variances) in the neural population. The constrained output power and the output noise together determine the ‘bandwidth’ of the system—that is, the expressive capacity of its outputs. Consider a neural population with input noise, output noise, and a fixed amount of output power. We find that when input signal variability is sufficiently large compared to the input noise, the gain of neurons should decrease with the variance of the input (regions to the right of the peaks in the right-hand panel of Figure 4A). This is a regime where the output bandwidth is low compared to the input range, and efficient coding predicts that signals should be ‘whitened’ by equalizing the variance in different channels. Conversely, consider input signals with a smaller range, which are thus more disrupted by input noise. In this case, the gain of neurons should increase with the variance of the input (regions to the left of the peaks in the right-hand panel of Figure 4A). This is a regime where the input noise dominates, and efficient coding predicts that the system should invest more resources in more variable, and hence more easily detectable, input signals. The relative sizes of input and output noise (controlled by $\text{Λ}$ in Figure 4A) determines the input ranges over which the two qualitatively different regimes of efficient coding apply.

Figure 4 with 4 supplements see all

Download asset Open asset

To make these abstract considerations concrete, we first considered coding in the sensory periphery. A common strategy employed in the periphery is ‘whitening’, where relatively fewer resources are devoted (yielding lower gain) to features with more variation (Olshausen and Field, 1996). As an example, within the spatial frequency range that the retina captures well, sensitivity is greater for high spatial frequencies than for low ones, that is, sensitivity is inversely related to the degree of variation in natural scenes (the well-known $1/f^2$ power spectrum [Olshausen and Field, 1996]). Figure 4B illustrates how this strategy can emerge from the simple efficient coding scheme discussed above as applied to peripheral sensory processing. Spatiotemporal correlations of light undergo filtering before passing through the optic nerve bottleneck (a constraint on bandwidth). Such a constraint on bandwidth is equivalently understood as a regime where output noise is relatively large compared to input noise. In this limit, where output noise dominates over input noise, the optimal strategy is whitening (See Srinivasan et al., 1982 and Figure 4A). Of course, real neural systems contend with both input and output noise; indeed recent work has shown that simply whitening to deal with output noise underestimates the optimal performance that the sensory periphery can achieve (Doi and Lewicki, 2014).

An alternative regime arises when input noise limits performance. In this regime, relatively more resources are devoted to features with more variation. This regime was discussed in early work of van Hateren, (1992a) and was also recognized in (Doi and Lewicki, 2011, 2014), although it has received much less attention than the ‘whitening’ regime. Our results suggest that this is the regime is relevant to cortex, where it predicts the relative allocation of resources to higher-order image statistics. Figure 4C illustrates the simple efficient coding scheme in this context. We use our image preprocessing pipeline to mimic early visual processing, and we consider the downstream coding of higher-order image features. Because these features must be sampled from a finite patch of an image, they are subject to input noise arising from fluctuations in statistical estimation. When such input noise is limiting, the ability to detect a signal from noise increases with the variability of that signal. In this limit, efficient coding predicts that resources should be allocated in proportion to feature variability (Figure 4C). This captures the intuition that when signal reliability is in question, more reliable signals warrant more resources. Furthermore, if two or more channels have covarying signals, resources should be devoted in relation to the direction and degree of maximum covariance (see ‘Materials and methods’, Two regimes of efficient coding, Figure 4—figure supplement 3, and Figure 4—figure supplement 4).

The difference between these two efficient coding regimes is a consequence of the form of noise—output vs input noise—that is limiting. Our finding that cortex operates in a different regime than the well-known peripheral whitening reflects the fact that different stages and kinds of processing can face different constraints. While information transmission by the visual periphery is limited by a bottleneck in the optic nerve, cortex faces no such transmission constraint. Furthermore, while faithful encoding may be an immediate goal of early visual processing, cortical circuits have to interpret image features from a complex and crowded visual scene and perform statistical inference. For example, to discriminate between various textures, the cortex cannot perform pixel-by-pixel comparisons, but must rely on the estimation of local correlations (image statistics) instead. Because these correlations must be sampled from a finite patch of the visual scene, any estimate will be limited by sampling fluctuations.

1. 1. Atick J
   2. Redlich AN

   (1990) Toward a theory of early visual processing

   Neural Computation 2:308–320.

   https://doi.org/10.1162/neco.1990.2.3.308

   • Google Scholar
2. 1. Atick J
   2. Li Z
   3. Redlich AN

   (1992) Understanding retinal color coding from first principles

   Neural Computation 4:449–572.

   https://doi.org/10.1162/neco.1992.4.4.559

   • Google Scholar
3. 1. Attneave F

   (1954) Some informational aspects of visual perception

   Psychological Review 61:183–193.

   https://doi.org/10.1037/h0054663

   • Google Scholar
4. 1. Balasubramanian V
   2. Sterling P

   (2009) Receptive fields and the functional architecture in the retina

   The Journal of Physiology 587:2753–2767.

   https://doi.org/10.1113/jphysiol.2009.170704

   • Google Scholar
5. 1. Barlow HB

   (1959)

   Sensory mechanisms, the reduction of redundancy, and intelligence

   537–574, Proceedings of the 17 Symposium on the Mechanization of Thought Processes, Vol. 2, HM Stationery Office, London.

   • Google Scholar
6. 1. Barlow HB

   (1961)

   Sensory Communication

   217–234, Sensory Communication, MIT Press, Cambridge.

   • Google Scholar
7. 1. Barlow H

   (2001) Redundancy reduction revisited

   Network 12:241–253.

   https://doi.org/10.1088/0954-898X/12/3/301

   • Google Scholar
8. 1. Bell AJ
   2. Sejnowski TJ

   (1997) The “independent components” of natural scenes are edge filters

   Vision Research 37:3327–3338.

   https://doi.org/10.1016/S0042-6989(97)00121-1

   • Google Scholar
9. 1. Borghuis BG
   2. Ratliff CP
   3. Smith RG
   4. Sterling P
   5. Balasubramanian V

   (2008) Design of a neuronal array

   Jhe Journal of Neuroscience 28:3178–3189.

   https://doi.org/10.1523/JNEUROSCI.5259-07.2008

   • Google Scholar
10. 1. Brenner N
    2. Bialek W
    3. de Ruyter van Steveninck R

    (2000) Adaptive rescaling maximizes information transmission

    Neuron 26:695–702.

    https://doi.org/10.1016/S0896-6273(00)81205-2

    • Google Scholar
11. 1. Brincat SL
    2. Connor CE

    (2004) Underlying principles of visual shape selectivity in posterior inferotemporal cortex

    Nature Neuroscience 7:880–886.

    https://doi.org/10.1038/nn1278

    • Google Scholar
12. 1. Campbell FW
    2. Kulikowski JJ
    3. Levinson J

    (1966)

    The effect of orientation on the visual resolution of gratings

    The Journal of physiology 187:427–436.

    • Google Scholar
13. 1. Carlson NL
    2. DeWeese MR

    (2002)

    Learning of sparse auditory receptive fields

    1103–1108, Proceedings of the 2002 International Joint Conference on Neural Networks, 2.

    • Google Scholar
14. 1. Carlson NL
    2. DeWeese MR

    (2012) Sparse codes for speech predict spectrotemporal receptive fields in the inferior colliculus

    PLOS Computational Biology 8:e1002594.

    https://doi.org/10.1371/journal.pcbi.1002594

    • Google Scholar
15. 1. Chubb C
    2. Landy MS
    3. Econopouly J

    (2004) A visual mechanism tuned to black

    Vision Research 44:3223–3232.

    https://doi.org/10.1016/j.visres.2004.07.019

    • Google Scholar
16. 1. Croner LJ
    2. Kaplan E

    (1995) Receptive fields of p and m ganglion cells across the primate retina

    Vision Research 35:7–24.

    https://doi.org/10.1016/0042-6989(94)E0066-T

    • Google Scholar
17. 1. Doi E
    2. Lewicki MS

    (2011) Characterization of minimum error linear coding with sensory and neural noise

    Neural Computation 23:2498–2510.

    https://doi.org/10.1162/NECO_a_00181

    • Google Scholar
18. 1. Doi E
    2. Lewicki MS

    (2014) Simple model of optimal population coding for sensory systems

    PLOS Computational Biology 10:e1003761.

    https://doi.org/10.1371/journal.pcbi.1003761

    • Google Scholar
19. 1. Fairhall AL
    2. Lewen GD
    3. Bialek W
    4. de Ruyter van Steveninck R

    (2001) Efficiency and ambiguity in an adaptive neural code

    Nature 412:787–792.

    https://doi.org/10.1038/35090500

    • Google Scholar
20. 1. Field DJ

    (1987) Relations between the statistics of natural images and the response properties of cortical cells

    Journal of the Optical Society of America A, Optics and Image Science 4:2379–2394.

    https://doi.org/10.1364/JOSAA.4.002379

    • Google Scholar
21. 1. Ganguli D
    2. Simoncelli EP

    (2011)

    Advances in Neural Information Processing Systems

    Advances in Neural Information Processing Systems, Vol. 23, MIT Press.

    • Google Scholar
22. 1. Garrigan P
    2. Ratliff CP
    3. Klein JM
    4. Sterling P
    5. Brainard DH
    6. Balasubramanian V

    (2010) Design of a trichromatic cone array

    PLOS Computational Biology 6:e1000677.

    https://doi.org/10.1371/journal.pcbi.1000677

    • Google Scholar
23. 1. Hung CC
    2. Carlson ET
    3. Connor CE

    (2012) Medial axis shape coding in macaque inferotemporal cortex

    Neuron 12:1099–1113.

    https://doi.org/10.1016/j.neuron.2012.04.029

    • Google Scholar
24. 1. Hyvarinen A
    2. Hoyer P

    (2000) Emergence of phase and shift invariant features by decomposition of natural images into independent feature subspaces

    Neural Computation 12:1705–1720.

    https://doi.org/10.1162/089976600300015312

    • Google Scholar
25. 1. Karklin Y
    2. Lewicki MS

    (2009) Emergence of complex cell properties by learning to generalize in natural scenes

    Nature 457:83–86.

    https://doi.org/10.1038/nature07481

    • Google Scholar
26. 1. Karklin Y
    2. Simoncelli EP

    (2001)

    Efficient coding of natural images with a population of noisy, non-linear neurons

    Advances in Neural Information Processing Systems 24:999–1007.

    • Google Scholar
27. 1. Kaschube M
    2. Schnabel M
    3. Lowel S
    4. Coppola DM
    5. White LE
    6. Wolf F

    (2011) Universality in the evolution of orientation columns in the visual cortex

    Science 330:1113–1116.

    https://doi.org/10.1126/science.1194869

    • Google Scholar
28. 1. Kremers J
    2. Lee BB
    3. Yeh T

    (1995)

    Receptive field dimensions of macaque retinal ganglion cells

    399–405, Colour Vision Deficiencies, Documenta Ophthalmologica Proceedings Series.

    • Google Scholar
29. 1. Kuang X
    2. Poletti M
    3. Victor JD
    4. Rucci M

    (2012) Temporal encoding of spatial information during active visual fixation

    Current Biology 22:510–514.

    https://doi.org/10.1016/j.cub.2012.01.050

    • Google Scholar
30. 1. Laughlin S

    (1981)

    A simple coding procedure enhances a neuron's information capacity

    Zeitschrift für Naturforschung C A journal of biosciences 36:910–912.

    • Google Scholar
31. 1. Lewicki MS

    (2002)

    Efficient coding of natural sounds

    Nature 5:356–363.

    • Google Scholar
32. 1. Liu YS
    2. Stevens CF
    3. Sharpee TO

    (2009) Predictable irregularities in retinal receptive fields

    Proceedings of the National Academy of Sciences of USA 106:16499–16504.

    https://doi.org/10.1073/pnas.0908926106

    • Google Scholar
33. 1. Morrone MC
    2. Burr DC

    (1988)

    Feature detection in human vision: a phase-dependent energy model

    Proceedings of the Royal Society of London Series B, Containing Papers of a Biological Character 235:221–245.

    • Google Scholar
34. 1. Nirenberg S
    2. Pandarinath C

    (2012) Retinal prosthetic strategy with the capacity to restore normal vision

    Proceedings of the National Academy of Sciences of USA 109:15012–15017.

    https://doi.org/10.1073/pnas.1207035109

    • Google Scholar
35. 1. Olshausen BA
    2. Field DJ

    (1996) Emergence of simple-cell receptive field properties by learning a sparse code for neural images

    Nature 381:607–609.

    https://doi.org/10.1038/381607a0

    • Google Scholar
36. 1. Olshausen BA
    2. Field DJ

    (1997) Sparse coding with an overcomplete basis set: a strategy employed by v1?

    Vision Research 37:3311–3325.

    https://doi.org/10.1016/S0042-6989(97)00169-7

    • Google Scholar
37. 1. Oppenheim AV
    2. Lim JS

    (1981) The importance of phase in signals

    Proceedings of the IEEE 69:529–541.

    https://doi.org/10.1109/PROC.1981.12022

    • Google Scholar
38. 1. Purpura K
    2. Victor JD
    3. Katz E

    (1994) Striate cortex extracts higher-order spatial correlations from visual textures

    Proceedings of the National Academy of Sciences of USA 91:8482–8486.

    https://doi.org/10.1073/pnas.91.18.8482

    • Google Scholar
39. 1. Ratliff CP
    2. Borghuis BG
    3. Kao Y-H
    4. Sterling P
    5. Balasubramanian V

    (2010) Retina is structured to process an excess of darkness in natural scenes

    Proceedings of the National Academy of Sciences of USA 107:17368–17373.

    https://doi.org/10.1073/pnas.1005846107

    • Google Scholar
40. 1. Ruderman DL
    2. Bialek W

    (1994)

    Statistics of natural images: scaling in the woods

    Physical Review Letters 73:814–817.

    • Google Scholar
41. 1. Ruderman DL

    (1997) Origins of scaling in natural images

    Vision Research 37:3385–3398.

    https://doi.org/10.1103/PhysRevLett.73.814

    • Google Scholar
42. 1. Schwartz O
    2. Simoncelli EP

    (2001) Natural signal statistics and sensory gain control

    Nature Neuroscience 4:819–825.

    https://doi.org/10.1038/90526

    • Google Scholar
43. 1. Simoncelli EP

    (2002) Vision and the statistics of the visual environment

    Current Opinion in Neurobiology 13:144–149.

    https://doi.org/10.1016/S0959-4388(03)00047-3

    • Google Scholar
44. 1. Smith EC
    2. Lewicki MS

    (2006) Efficient auditory coding

    Nature 439:978–982.

    https://doi.org/10.1038/nature04485

    • Google Scholar
45. 1. Srinivasan MV
    2. Laughlin SB
    3. Dubs A

    (1982) Predictive coding: a fresh view of inhibition in the retina

    Poceedings of the Royal Society of London Series B, Containing papers of a Biological character 216:427–459.

    https://doi.org/10.1098/rspb.1982.0085

    • Google Scholar
46. 1. Stephens GJ
    2. Mora T
    3. Tkacik G
    4. Bialek W

    (2013) Statistical thermodynamics of natural images

    Physical Review Letters 110:018701.

    https://doi.org/10.1103/PhysRevLett.110.018701

    • Google Scholar
47. 1. Swindale NV
    2. Shoham D
    3. Grinvald A
    4. Bonhoeffer T
    5. Hubener M

    (2000) Visual cortex maps are optimized for uniform coverage

    Nature Neuroscience 3:822–826.

    https://doi.org/10.1038/77731

    • Google Scholar
48. 1. Tkačik G
    2. Prentice JS
    3. Victor JD
    4. Balasubramanian V

    (2010) Local statistics in natural scenes predict the saliency of synthetic textures

    Proceedings of the National Academy of Sciences of USA 107:18149–18154.

    https://doi.org/10.1073/pnas.0914916107

    • Google Scholar
49. 1. Tkačik G
    2. Ratliff CP
    3. Milčinski G
    4. Klein JM
    5. Seyfarth LH
    6. Sterling P
    7. Brainard DH
    8. Balasubramanian V

    (2011) Natural images from the birthplace of the human eye

    PLOS ONE 6:e20409.

    https://doi.org/10.1371/journal.pone.0020409

    • Google Scholar
50. 1. Torralba A
    2. Oliva A

    (2003) Statistics of natural image categories

    Network 14:391–412.

    https://doi.org/10.1088/0954-898X/14/3/302

    • Google Scholar
51. 1. van Hateren JH
    2. Ruderman DL

    (1998)

    Independent component analysis of natural image sequences yields spatiotemporal filters similar to simple cells in primary visual cortex

    Proceedings of the Royal Society of London Series B, Containing papers of a Biological character 265:2315–2320.

    • Google Scholar
52. 1. van Hateren JH
    2. van der Schaaf A

    (1998)

    Independent component filters of natural images compared with simple cells in primary visual cortex

    Proceedings Biological sciences/The Royal Society 265:359–366.

    • Google Scholar
53. 1. van Hateren JH

    (1992a) A theory of maximizing sensory information

    Biological Cybernetics 68:23–29.

    https://doi.org/10.1007/BF00203134

    • Google Scholar
54. 1. van Hateren JH

    (1992b) Theoretical predictions of spatiotemporal receptive fields of fly lmcs, and experimental validation

    Journal of Comparative Physiology A 171:157–170.

    https://doi.org/10.1007/BF00188924

    • Google Scholar
55. 1. Victor JD
    2. Conte MM

    (1991) Spatial organization of nonlinear interactions in form perception

    Vision Research 31:1457–1488.

    https://doi.org/10.1016/0042-6989(91)90125-O

    • Google Scholar
56. 1. Victor JD
    2. Conte MM

    (2005) Local processes and spatial pooling in texture and symmetry detection

    Vision Research 45:1063–1073.

    https://doi.org/10.1016/j.visres.2004.10.012

    • Google Scholar
57. 1. Victor JD
    2. Conte MM

    (2012) Local image statistics: maximum-entropy constructions and perceptual salience

    Journal of the Optical Society of America A, Optics, Image Science, and Vision 29:1313–1345.

    https://doi.org/10.1364/JOSAA.29.001313

    • Google Scholar
58. 1. Victor JD
    2. Chubb C
    3. Conte MM

    (2005) Interaction of luminance and higher-order statistics in texture discrimination

    Vision Research 45:311–328.

    https://doi.org/10.1016/j.visres.2004.08.013

    • Google Scholar
59. 1. Victor JD
    2. Thengone DJ
    3. Conte MM

    (2013) Perception of second- and third-order orientation signals, and their interactions

    Journal of Vision 13:1–21.

    https://doi.org/10.1167/13.4.21

    • Google Scholar
60. 1. Victor JD

    (1986) Isolation of components due to intracortical processing in the visual evoked potential

    Proceedings of the National Academy of Sciences of USA 83:7984–7988.

    https://doi.org/10.1073/pnas.83.20.7984

    • Google Scholar
61. 1. Vinje WE
    2. Gallant JL

    (2000) Sparse coding and decorrelation in primary visual cortex

    Science 287:1273–1276.

    https://doi.org/10.1126/science.287.5456.1273

    • Google Scholar
62. 1. von der Heydt R
    2. Peterhans E
    3. Baumgartner G

    (1984) Illusory contours and cortical neuron responses

    Science 224:1260–1262.

    https://doi.org/10.1126/science.6539501

    • Google Scholar
63. 1. Wolf F
    2. Geisel T

    (1998) Spontaneous pinwheel annihilation during visual development

    Nature 395:73–78.

    https://doi.org/10.1038/25736

    • Google Scholar
64. 1. Yau JM
    2. Pasupathy A
    3. Brincat SL
    4. Connor CE

    (2012) Curvature processing dynamics in macaque area v4

    Cerebral Cortex 23:198–209.

    https://doi.org/10.1093/cercor/bhs004

    • Google Scholar
65. 1. Yu Y
    2. Schmid AM
    3. Victor JD

    (2013)

    The laminar origin of sensitivity to high-order image statistics in macaque visual cortex

    Proceedings of the CoSyNe Annual Meeting.

    • Google Scholar

Thank you for sending your work entitled “Variance is salience: efficient coding in central sensory processing” for consideration at eLife. Your article has been evaluated by Eve Marder (Senior editor), Timothy Behrens (Reviewing editor), and 3 reviewers.

The Reviewing editor and the other reviewers discussed their comments before we reached this our decision, and the Reviewing editor has assembled the following comments to help you prepare a revision.

All reviewers found the data interesting, but all had concerns about whether they addressed the central hypotheses outlined in the abstract and introduction. In brief we do not find a convincing argument that the results 1) reflect a central mechanism, 2) relate to behavior, or 3) make different predictions than what one would predict for the sensory periphery given the efficient coding hypothesis. These questions are specified in detail in the reviews below. The consensus that emerged from the consultation session is that clarification and rewriting will be needed to ensure that the contribution of this work can be understood. Ordinarily eLife does not provide the full reviews, but in this case we believe it is important that you see the concerns of the reviewers as you prepare your revision.

Reviewer #1

The authors show that psychophysical detection thresholds for figure/ground segmentation of image patches in noise are extremely well predicted by the extent to which the features defining the patch are variable in natural images. Both follow a characteristic pattern in which cardinal, then oblique features are the most variable over images (and the most detectable), followed by fourth order statistics, followed by third order (L shapes in a 4-pixel glider). The correspondence between lines of isosensitivity and the precision matrix for pairwise comparisons in 10 dimensions are very striking. The authors interpret their findings as showing that efficient coding - heightened sensitivity to the most informative features of the world - occurs not only at a sensory level but at a central decision level.

I liked the paper and approach very much. I was a little more doubtful about the novelty of the findings. My concerns are as follows:

1) I wasn't sure how the paper demonstrated that efficient coding operates at a 'central' level, as the authors claim in the abstract In fact, the paper doesn't really have anything to do with what most cognitive scientists think of as 'central processing'. It shows that we are most sensitive to patches defined by cardinal orientations, and that variability along this axis is an informative way to characterise natural scenes.

2) The psychophysical results seems in part to restate the well-known oblique effect. It is known that we are most sensitive to cardinal orientations, and the neural underpinnings of this effect (in visual cortex) have been much discussed. The work of Simoncelli and colleagues (cited appropriately here) has been key in pointing out that sensitivity to cardinal directions matches the prevalence of cardinal information in the world. I think the work here extends that finding by demonstrating a particularly good correspondence between the spatial correlations in image statistics and this sensitivity over a number of basic features.

In short, I think this is an interestingly, well-presented paper that I would encourage others to read, and that I would cite in my own work. However, I am not sure whether it is telling us something we didn't know, or just demonstrating something we did know in a particularly elegant fashion.

Reviewer #2

This paper describes a robust and interesting observation that links properties of natural images to psychophysical properties of the visual system. While this observation is certainly worthwhile publishing, I'm less convinced about the author's conclusion with regard to “variance is salience” as an efficient coding principle for central sensory processing.

While the authors correctly note that central processing must be concerned with behaviorally relevant aspects of the inputs, the 'variance' in their principle is completely independent of the organism's behavior and purely defined in terms of the stimulus. In fact, none of their discussion nor references to the earlier van Hateren papers that the authors invoke as a framework for interpreting their findings involves behavior. I suggest to simply drop that part of their claim and the suggestion that their principle might account for central processing and describe it as a principle governing early sensory processing where different constraints apply than in the sensory periphery. (Unless I'm missing a central part of their argument, of course. Example below.)

I think the paper would benefit greatly from a more explicit setup, ideally including a Figure, of the various potential constraints faced by early sensory processing and their respective implications for their experiments. While this aspect is not the main contribution of their paper, it would make it more explicit which alternative models/constraints are being excluded by their observation. It would also delineate better, what is inevitable conclusion within the efficient coding framework (finite resources and high input noise), and what is interpretation/speculation. Both are currently mixed in the Discussion section.

Psychophysical measurements necessarily characterize the entire system; from periphery to behavior. The author suggest their results support an optimization principle separate and different from that in the periphery and while I agree on an intuitive level, I would welcome help in thinking through the implications of combining two different principles, applied after each other, for the aggregate quantities that are being measured. E.g. why does the high input noise constraint only refer to what enters cortex, not already the retina, overriding the output-bottleneck in the visual periphery?

Finally, I'd like to commend the authors on making excellent use of the possibility of including supplementary information.

In summary, I recommend publication of this paper subject to:

1) Providing a more explicit setup for the different hypotheses to be tested and how they derive from the efficient coding principle subject to different constraints.

2) Omitting the behaviorally relevant claims not supported by their framework.

3) Responses to more minor points of critique below.

Reviewer #3

The authors demonstrate that the level of variability in various local, multi-point statistics in natural scenes quantitatively predicts the relative salience of these statistics. I find their data largely convincing, and I feel that this is an important question and that the authors are careful and creative in their approach here. However, I have several concerns, which I suspect in part reflect my own ignorance or misunderstanding about the present paper and past work by these authors.

First, I am not clear on what is qualitatively new here beyond what was shown in the nice 2010 PNAS paper by several of the same authors. I can see that some of the details of their analyses are different, and the comparisons here are more quantitative, but it seems to me that the main ideas and conclusions were already in place in the previous work.

The authors pitch their results in terms of cortical processing, but the only data presented are from human psychophysics, used here as a proxy for cortical processing. The authors give arguments for why salience strongly reflects cortical processing. But I feel that the claims about cortical processing are stated too strongly in some parts of the paper given that they do not directly measure cortical activity nor perform any manipulations to cortex, for example.

It seems to me that the dichotomy set up by the authors between predictions for peripheral and central processing is somewhat overstated. For example, in the final sentence of the Abstract, they state that the efficient coding hypothesis applies in “a different guise” and makes “different predictions” for the central nervous system than the periphery. They predict that sensitivity in the CNS should be greater for highly variable stimuli, but it is not clear to me that this is not one of the predictions one would make for peripheral sensory neurons using this principle.

What was the exact task being performed by the subjects?

What would Figure 1E look like for IId samples from white (or colored) noise?

How do the results change for images pixelated along a grid that is 45° tilted relative to the vertical and horizontal directions?

Related to this, could the √2 differences in the degree of variation for the vertical/horizontal and diagonal beta data points in Figure 3A be due to the difference in the distance between the points in question given the orientation of the pixelation grid?

All reviewers found the data interesting, but all had concerns about whether they addressed the central hypotheses outlined in the abstract and introduction. In brief we do not find a convincing argument that the results 1) reflect a central mechanism, 2) relate to behavior, or 3) make different predictions than what one would predict for the sensory periphery given the efficient coding hypothesis. These questions are specified in detail in the reviews below. The consensus that emerged from the consultation session is that clarification and rewriting will be needed to ensure that the contribution of this work can be understood. Ordinarily eLife does not provide the full reviews, but in this case we believe it is important that you see the concerns of the reviewers as you prepare your revision.

Response to broad concerns:

1) To what extent do our results reflect a central mechanism?

There are three lines of evidence, and we detail these below and in the manuscript. First, stimuli are all high contrast (100%) and of a readily visible size (14 arcmin), so retinal limitations of contrast sensitivity and resolution are eliminated. Second, the task requires pooling of information over wide areas. Third, extracting three- and four-point correlations requires a kind of nonlinear processing (two stages of nonlinearity) that is not generally considered to be present in the retina; physiologic recordings show that neural responses to these correlations are not present in the thalamus, but are present in visual cortex, and much more so in V2 than in V1. We acknowledge that these lines are indirect; we do not directly record cortical activity, and we now state this explicitly in the manuscript (see response to comments raised by Reviewer 3). But we believe that together, these lines of evidence justify the statement that the thresholds are determined by limitations of central processing.

2) To what extent do our results relate to behavior?

The goal of the visual system is to make discriminations that are useful to guide action, rather than to reconstruct the image per se. We had used the term “behaviorally relevant” to highlight this idea, once in the Abstract and once in the manuscript; we have now removed it from the Abstract, where we see how it might have been confusing, and we expand on the point in the Introduction.

With regard to the comments raised by Reviewer 2, obviously our analysis of natural image statistics was without regard to task, and the psychophysical task was not designed to mimic a real-life behavior. But we think this is a strong point, not a weakness, as it shows that the allocation of resources in early cortical visual processing can be accounted for in a general framework. We now comment on this in the text.

3) To what extent do our results make different predictions than one would predict for the sensory periphery?

We now clarify this at several points in the manuscript, and we add a new Figure. Briefly, the contrast is not between peripheral and central processing per se, but the difference between two efficient coding regimes: one that is limited by noise in the output (i.e., limited capacity) and one regime that is limited by noise in the input (i.e., limited sampling). The periphery is typically considered to be characterized by the first regime, at least when it comes to thinking about receptive field properties, and we hypothesize that central processing is characterized by the second. The new Figure illustrates the qualitative difference between these regimes: when output noise is limiting, efficient coding predicts a whitening (reduced sensitivity for more variable input components); when input noise is limiting, efficient coding predicts the opposite (increased sensitivity for more variable input components).

Since central processing follows the peripheral processing, one could wonder how the two stages of the same processing stream can be subject to different constraints. The reason is as follows: the “input signal” in the context of efficient coding for the retina is directly the raw light intensity, and efficient coding regime is derived from the spatiotemporal correlation structure of light intensity. In contrast, the input signals for central coding are nonlinear functions of the original image (image statistic coordinates); this means that efficient coding regime is derived from the covariance structure of image statistics. Since the formal inputs to the encoders are very different (light or contrast in the periphery, image statistic estimates centrally), a different regime of efficient coding emerges.

Reviewer #1

[…] In short, I think this is an interestingly, well-presented paper that I would encourage others to read, and that I would cite in my own work. However, I am not sure whether it is telling us something we didn't know, or just demonstrating something we did know in a particularly elegant fashion.

We are very pleased that Reviewer 1 found our paper to be both interesting and well-presented. We do feel that the paper is telling us something new and rather exciting, as we explain below in responses to the specific comments.

1) I wasn’t sure how the paper demonstrated that efficient coding operates at a ‘central’ level, as the authors claim in the abstract. In fact, the paper doesn’t really have anything to do with what most cognitive scientists think of as ‘central processing’.

With regard to ‘central processing’: We use this term to distinguish our focus – cortical sensory processing – from processing in the sensory periphery. We think this is a reasonable use of the term, even though there is (of course) much additional central processing that we do not consider. But we understand the potential for confusion, and we now clarify the use of the term in the Introduction: “To test this hypothesis, we focus on early stages of central visual processing. Here, early visual cortex (V1 and V2) is charged with extracting edges, shapes, and other complex correlations of light between multiple points in space...”

We feel that there is strong evidence that the computations that determine task performance occur in primary visual cortex, both V1 and V2, and we expand on this point in the paper. Specifically, we now state:

Discussion: “Although we did not record cortical responses directly, several lines of evidence indicate that that the perceptual thresholds we measured are determined by cortical processes. […] Conversely, macaque visual cortical neurons (Purpura et al., 1994), especially those in V2, manifest responses to three- and four-point correlations (Yu et al., 2013). ”

It shows that we are most sensitive to patches defined by cardinal orientations, and that variability along this axis is an informative way to characterise natural scenes.

Our results about sensitivities to pairwise statistics in horizontal, vertical, and diagonal directions go substantially beyond a demonstration of previously-known findings, in several ways.

First, the “on-axis” sensitivities are not obvious consequences of the orientation tuning of cortical neurons (since we consider both positive and negative correlations) or the oblique effect (our observed differences are substantially larger, likely because these configurations differ in more than orientation). For further discussion of the oblique effect, please see our response to point 2 below.

Second, only a small portion of our findings (a prediction of four relative sensitivities) relate to individual “axes.” Most of our findings (more than a dozen free parameters) relate to how the axes interact – again, not something that is a consequence of orientation-tuning or the oblique effect.

Finally, it is possible that there is a misunderstanding about the axes themselves: they are not the spatial axes within an image, but abstract axes within a space of image statistics. We can see how this might have been confusing, and we now clarify it with the following text:

Results: “When pairs of natural image statistics covary, thus sampling oblique directions not aligned with the coordinate axes in the space of image statistics, our hypothesis predicts that human perceptual sensitivity is matched to both the degree and the direction of that covariation (we are referring here to the orientation of a distribution in the coordinate plane of a pair of image statistics, and not to an orientation in physical space).”

An important way in which our results extend beyond what is already known relates to the predictions of normative theories. To date, normative theories applied to cortical processing have been focused on predicting the response properties of single cells, considered as linear or quasilinear filters. Here, we ask how resources should be distributed across a population of cells to represent a diverse set of nonlinear, higher-order features that cover a multidimensional domain. We highlight this goal within the Introduction:

Introduction: “Can we extend such theories beyond the sensory periphery to describe cortical sensitivity to complex sensory features? […] We will show that these ideas predict a specific organizing principle for aggregate sensitivities arising in cortex: the perceptual salience of complex sensory signals increases with the variability, or unpredictability, of the corresponding signals over the ensemble of natural stimuli.”

We additionally emphasize that the goal of our study is to measure how sensitivities are distributed across a set of features, and is not limited to identifying features for which sensitivity is greatest. We now highlight this within the Introduction and Discussion:

Introduction: “We compare the spatial variation of local patterns of light across natural images with human sensitivity to manipulations of the same patterns in synthetic images. This allows us to determine how sensitivity is distributed across many different features, rather than simply determining the most salient ones. To this end...”

Discussion: “How should neural mechanisms be distributed to represent a diverse set of informative sensory features? We argued that...”

2) The psychophysical results seems in part to restate the well-known oblique effect. It is known that we are most sensitive to cardinal orientations, and the neural underpinnings of this effect (in visual cortex) have been much discussed. The work of Simoncelli and colleagues (cited appropriately here) has been key in pointing out that sensitivity to cardinal directions matches the prevalence of cardinal information in the world. I think the work here extends that finding by demonstrating a particularly good correspondence between the spatial correlations in image statistics and this sensitivity over a number of basic features.

While we can see how our findings might at first glance be considered to be part of an “oblique effect,” they are actually quite different. The effect size we report is much larger: sensitivities are about 50% higher for cardinal directions vs diagonal ones; in the classical oblique effect, sensitivity for oblique- vs cardinal-direction gratings differ by 10–20% in the midrange of spatial frequencies (Campbell et al., 1966). The reason that the effect size we observe is much larger is that the horizontal and vertical pairwise correlations differ from the diagonal pairwise correlations in more ways than just orientation: checks involved in horizontal and vertical pairwise correlations share an edge, while checks involved in diagonal pairwise correlations only share a corner. Furthermore, the image property relevant to explaining the oblique effect in the work of Simoncelli and colleagues are the local statistics of image gradients / oriented edges. While pairwise correlations between pixels (that we study) can influence image gradients (responsible for explaining the oblique effect) and vice versa, there is no simple one-to-one mapping between the two sets of statistics (e.g., high beta-vertical does not by itself imply a lot of vertical edge fragments). We now explain how our findings differ from the oblique effect:

Results: “Note that the difference between the sensitivities in the horizontal and vertical directions (β− and β|) vs the diagonal directions (β\ and β/) is not simply an “oblique effect”, i.e., a greater sensitivity for cardinally- vs obliquely-oriented contours (Campbell et al., 1966). Horizontal and vertical pairwise correlations differ from the diagonal pairwise configurations in more than just orientation: checks involved in horizontal and vertical pairwise correlations share an edge, while checks involved in diagonal pairwise correlations only share a corner. Correspondingly, the difference in sensitivities for horizontal and vertical correlations vs diagonal correlations is approximately 50%, which is much larger than the size of the classical oblique effect (10–20%) (Campbell et al., 1966).”

Additionally, much of our study is devoted to third and fourth-order correlations and the interactions of the texture statistics. These statistics are qualitatively different from any pairwise correlations that might be relevant for the oblique effect, and are crucial for our demonstration that efficient coding also applies beyond the pairwise order.

In sum, we agree that the main finding of the work can be framed in the context of the match between image statistics and visual sensitivities (as in the Simoncelli framework), but we think that it is important to emphasize that we are not looking at an “oblique effect.”

Reviewer #2

[…] In summary, I recommend publication of this paper subject to:

1) Providing a more explicit setup for the different hypotheses to be tested and how they derive from the efficient coding principle subject to different constraints.

2) Omitting the behaviorally relevant claims not supported by their framework.

3) Responses to more minor points of critique below.

We are very pleased that Reviewer 2 recommends publication of our paper subject to his/her outlined changes. Below, we address each of his/her concerns: (A) is addressed in points 1-2 below, (B) is addressed in points 3 and 9 below, and (C) is addressed in points 4-8 below.

1) I think the paper would benefit greatly from a more explicit setup, ideally including a figure, of the various potential constraints faced by early sensory processing and their respective implications for their experiments. While this aspect is not the main contribution of their paper, it would make it more explicit which alternative models/constraints are being excluded by their observation. It would also delineate better, what is inevitable conclusion within the efficient coding framework (finite resources and high input noise), and what is interpretation/speculation. Both are currently mixed in the Discussion section.

We agree. We added substantial clarifying material and a new figure (Figure 4), as described below.

2) Psychophysical measurements necessarily characterize the entire system - from periphery to behavior. The authors suggest their results support an optimization principle separate and different from that in the periphery and while I agree on an intuitive level, I would welcome help in thinking through the implications of combining two different principles, applied after each other, for the aggregate quantities that are being measured. E.g. why does the high input noise constraint only refer to what enters cortex, not already the retina, overriding the output-bottleneck in the visual periphery?

If we understand the reviewer’s comment, it can be rephrased as follows: how can the cortex be operating in a regime dominated by input noise, when it follows peripheral processing, which sees the same input but is dominated by output noise?

The resolution of this apparent paradox is twofold. First, the critical issue is not the amount of input noise, but rather its size relative to the output noise. So, even though there is input noise at the level of the visual input, its severe output restriction (the transmission bottleneck) means that peripheral processing may be operating in a regime dominated by output noise. For cortex, which does not have the same output bottleneck, the same amount of input noise can dominate. The second consideration is the following. As we and others suggest, we view the “job” of early cortical processing as making inferences about surfaces and materials from visual texture (an idea that we formalize by measuring sensitivity to local image statistics). Given this goal, the fact that cortex only has access to a limited sample of a texture (i.e., the sample presented by a particular object) is also a form of input noise.

To illustrate this point, consider the following “toy visual system” whose task is to detect and estimate the local coherent motion of small, randomly moving point-like objects that are of high contrast (for example, the classic task of Newsome and Pare (1988), but with the dots placed on a 1/f background). Neurons in the periphery efficiently code the dots, making use of decorrelation. Centrally, however, the object of interest is the correlation structure of moving dots in small image patches, to extract the local coherent motion component. For this task, central processes must compare counts of how many dots move in each direction; at low density of dots, these estimates will be subject to large sampling errors due to random arrival and departure of dots from the region that is being sampled. Importantly, no matter what the SNR of the input is (even in 100% contrast vision where each dot is perfectly resolved), the local coherence estimates will be noisy due to the fact that local dot detections come as rare (and random) events; and the efficient central processing will thus be subject to large input noise. In our case, the local image statistics play the role of local dot counts, but the basic idea, that sampling is a form of noise, is identical. In both cases, because the signals relevant to periphery and cortex are different, one can end up in a different efficient coding regime at each stage of visual processing.

As suggested by the reviewer, we have added a new figure (Figure 4) to help clarify these points. Figure 4 illustrates the emergence of two efficient coding regimes in a system that operates under a total power (resource) constraint. The relevant regime, which depends on the relative strengths of input and output noise, reflects the constraints faced by the system in consideration. For our system (modeled as in Figure 4C), we take inputs to be nonlinear image features that have already undergone peripheral preprocessing, as caricatured by our image analysis pipeline. We then apply the efficient coding framework to this set of nonlinear features, and we ask how sensitivities should be distributed among them.

The key point is that because these image features (counts of glider colorings) must be sampled from a spatial region of an image, any estimation of these features will be limited by sampling noise. We hypothesize that this sampling noise, which is a form of input noise, dominates. This predicts (as shown in Figure 4), that sensitivity should increase with signal variability. We test this prediction and show that it holds.

We reorganized and added to the text in order to illustrate this:

Discussion: “A simple model of efficient coding by neural populations is shown in Figure 4A (details in Methods, Two regimes of efficient coding). Here, to enable analytical calculations, we used linear filters of variable gain and subject to Gaussian noise to model a population of neural channels encoding different features. […] The relative sizes of input and output noise (controlled by Λ in Figure 4A) determines the input ranges over which the two qualitatively different regimes of efficient coding apply.”

In addition, we clarified our language throughout the Discussion (see Discussion, Cortex faces a different class of challenges than the periphery ) to emphasize that the efficient coding regimes are a general feature of information optimization, and neither regime is a priori restricted to peripheral vs central processing. Thus, even in sensory periphery, there exist specific scenarios in which the signal is input- noise dominated. The best known example is perhaps that of night vision, when photon shot noise is no longer negligible and qualitatively reshapes peripheral information processing. Instead of decorrelation by center-surround receptive fields, receptive fields average the signal, i.e., they apply gain that enhances the correlations already present in the input in order to fight the detrimental effects of input noise.

We would like to emphasize clearly that the constraints considered here (nonlinear image features subject to sampling noise) thus differ from those considered by some of the most successful applications of efficient coding in the periphery. Because of these constraints, our finding that sensitivity increases with signal variability differs qualitatively from the findings of these studies, where whitening is optimal (or near optimal, see Doi and Lewicki (2014)) and sensitivity decreases with signal variability. Moreover, since feature extraction is thought to be one of the main tasks of central vision, the sampling noise constraint is likely to represent a general aspect of cortical processing, rather than a special case as for night vision in the periphery. And finally, the predictions made are quite different; we predict sensitivity to a wide variety of nonlinear features and their combinations, not the effective filtering behavior (i.e., shape) of a quasilinear receptive field.

3) While the authors correctly note that central processing must be concerned with behaviorally relevant aspects of the inputs, the ‘variance’ in their principle is completely independent of the organism’s behavior and purely defined in terms of the stimulus. In fact, none of their discussion nor references to the earlier van Hateren papers that the authors invoke as a framework for interpreting their findings involves behavior. I suggest to simply drop that part of their claim and the suggestion that their principle might account for central processing and describe it as a principle governing early sensory processing where different constraints apply than in the sensory periphery. (Unless I’m missing a central part of their argument, of course. Example below.)

We don’t intend to suggest that we are studying behavior, and we have edited the manuscript so that this is completely clear. We also added material to the Introduction and Discussion, including a new figure, that frames the work as consequences of differing constraints on peripheral and central processing.

Even though our focus is on visual processing and perception, the fact that visual processing must ultimately guide action plays a fundamental role: it motivates the idea that the visual system needs to make useful distinctions about the outside world, rather than reconstruct the image. But we can see that the term “behaviorally relevant” might have been misleading, so we have removed the term from the abstract, and we have explained the idea more carefully in the introduction:

“Can we extend such theories beyond the sensory periphery to describe cortical sensitivity to complex sensory features? Normative theories have been successful in predicting the response properties of single cells, including receptive fields in V1 (Olshausen and Field, 1996; Bell and Sejnowski, 1997; van Hateren and van der Schaaf, 1998; van Hateren and Ruderman, 1998; Hyvarinen and Hoyer, 2000; Vinje and Gallant, 2000; Karklin and Lewicki, 2009) and spectro-temporal receptive fields in primary auditory cortex (Carlson and DeWeese, 2002, 2012). […] We will show that these ideas predict a specific organizing principle for aggregate sensitivities arising in cortex: the perceptual salience of complex sensory signals increases with the variability, or unpredictability, of the corresponding signals over the ensemble of natural stimuli.”

Also (as mentioned in the response to the editors), we think that the fact that the image analysis and psychophysics was without regard to task is a strong point, not a weakness. We find it quite remarkable that we see such a striking match between natural image statistics and perceptual sensitivity without including any notion of “value” as related to behavior. This finding, that higher order percepts emerge from efficient coding without explicit specification of the goal, was the original hope of the efficient coding framework as first put forward by Barlow and Attneave (Barlow, 1959, 1961; Attneave, 1954). It remains an interesting direction for future research to ask how the inclusion of value would shape the framework studied here. We now mention this explicitly in the Discussion:

“Finally, we emphasize that although we have focused on perception of image statistics, we do this with the premise that this process is in the service of inferring the materials and objects that created an image and ultimately, guiding action. Thus, it is notable that we found a tight correspondence between visual perception and natural scene statistics without considering a specific task or behavioral set, and indeed, the emergence of higher order percepts without explicit specification of a task was the original hope of the efficient coding framework as first put forward by Barlow and Attneave (Barlow, 1959, 1961; Attneave, 1954). Doubtless, these “top-down” factors influence the visual computations that underlie perception, and the nature and site of this influence are an important focus of future research.”

4) In my understanding, the black/white asymmetry is the main systematic deviation from the prediction that was empirically found. I think this should be included (with Figure) and discussed in the main manuscript and not just in the SI.

We feel that the reviewer might have misunderstood our results in this regard: the main systematic deviation from prediction concerns positive and negative values of the fourth-order statistic α, not black/white asymmetry (the latter was eliminated by our preprocessing, in which we binarize image patches at the pixel intensity median such that each patch has equal numbers of black and white pixels (see Results, Analyzing local image statistics in natural scenes).

Specifically, there is an asymmetry of the distribution of α-values extracted from natural images that is not mirrored in psychophysical sensitivities. We mention this in Discussion, Clues to neural mechanisms, where we reference the relevant figure (Figure 3–figure supplement 8). We now also include the discussion of this asymmetry within the main manuscript, in Methods, Asymmetries in distributions of natural image statistics.

5) If your “glider” is anything more than a 2 × 2 window, then please explain in more detail.

The reviewer is correct: the glider is a 2 × 2 window. We now clarify this in two places in the main text:

Results: “As we recently showed, some informative local correlations of natural scenes are captured by the configurations of luminances seen through a “glider”, i.e., a window defined by a 2 × 2 square arrangement of pixels (Tkačik et al., 2010).”

Results: “We characterize each patch by the histogram of 16 binary colorings (2^2×2) seen through a square 2 × 2 pixel glider.”

6) Block average should be explained better in the main text; wouldn’t have understood it without resorting to SI.

We now clarify the block-averaging process within the main text:

Results: “We preprocess the image patches as shown in Figure 1A. This involves first averaging pixel luminances over a square region of N × N pixels, which converts an image of size L₁ × L₂ pixels into an image of reduced size L₁/N × L₂/N pixels. Images are then divided into R × R square patches of these downsampled pixels and whitened (see Methods, Image preprocessing, for further details).”

7) “Single scaling parameter for each image analysis”: what exactly does that mean? How many parameters per subject and comparison are needed? Can you say anything interesting about their distribution and the correlations between them?

We clarify these points in the main text, including a rewording of the quoted phrase. As we explain, we carried out image analyses for several choices of two size parameters: a block-average factor N (where N × N image pixels are averaged at the first step of the processing pipeline) and a patch size R (where R × R is the size of a patch in N × N pixels in which statistics are determined). There were 2 choices of N (2 and 4) and 3 choices of R (32, 48, and 64), for a total of 6 parallel image analyses (in the SI, we show that our results hold over a wider range too). For each of these analyses, there was just one scale factor (the same for all statistics and all subjects), adjusted to minimize the least-squares error between the set of standard deviations and the set of psychophysical sensitivities:

Results: “This qualitative comparison can be converted to a quantitative one (Figure 3A), as a single scaling parameter aligns the standard deviation of natural image statistics with the corresponding perceptual sensitivities. In this procedure, each of the six image analyses is scaled by a single multiplicative factor that minimizes the squared error between the set of standard deviations and the set of subject-averaged sensitivities (see Methods, Image preprocessing, and Figure 3–figure supplement 1 for additional details regarding scaling).”

With regard to a relationship between the scale factors and the analysis parameters (the block-average factor N and patch size R), there is (as one might expect) a systematic relationship. We now show this in detail in Methods, Image preprocessing, with a new figure and new text:

Methods: “To compare between natural image and psychophysical analyses, we scale the set of 9 standard deviations extracted from a given image analysis by a multiplicative factor that minimizes the squared error between the set of 9 standard deviations and the set of 9 psychophysical sensitivities. Figure 3–figure supplement 1 shows the value of the scale factor for different choices of the block-average factor N and patch size R.”

Figure 3–figure supplement 1. Scaling of natural image analyses. We scale each image analysis by a single scale factor that minimizes the squared error between the set of 9 standard deviations and the set of 9 psychophysical sensitivities. The scale factors are shown here as a function of block-average factor N for different choices of the patch size R. We find that the variance of image statistics decreases with increasing values of N, and thus larger values of N require a larger scale factor. Similarly, for a given value of N, the variance of image statistics increases with increasing R, and thus larger values of R require a larger scale factor.

8) I’d welcome it if the authors could spell out the Null-hypothesis for their significance tests, rather than saying what they did mechanistically (”coordinate labels independently shuffled”).

We now do this within the main text by replacing the mechanistic description of the significance tests with a description of each null hypothesis:

Results: “This value ranges from 0.987 to 0.999 across image analyses and was consistently larger than the value measured under the null hypothesis that the apparent correspondence be- tween statistics and sensitivities is chance (p ≤ 0.0003 for each image analysis; see Methods, Permutation tests, for details regarding statistical tests).”

Results: “This value, averaged across coordinate planes, ranges from 0.953 to 0.977 across image analyses. We compared this correspondence to that obtained under the null hypotheses that either (i) the apparent correspondence between image statistic covariances and isodiscrimination contours is chance, or (ii) the apparent covariances in image statistics are due to chance. The observed correspondence is much greater than the value measured under either null hypothesis (p ≤ 0.0003 for each image analysis under both hypotheses; see Methods, Analysis of image statistics in pairwise coordinate planes, and Figure 3–figure supplement 2 for comparisons of eccentricity and tilt, and Methods, Permutation tests, for statistical tests).”

We also made corresponding additions to Methods, Permutation tests, in order to supplement the mechanistic description of the significance tests with the full descriptions of the null hypotheses.

9) Discussion: “Cortical mechanisms should be...”: This is in clear contradiction to the behaviorally relevant argument elsewhere in the paper. Very variable features may be behaviorally relevant demanding cortical resources, or they may not be asking for them to be factored/normalized out (e.g. overall brightness). The theoretical framework is all about maximizing information with respect to the inputs, not with respect to behaviorally relevant outputs.

We agree, this sentence did not properly explain our ideas. We rewrote this section as follows:

Discussion: “How should neural mechanisms be distributed to represent a diverse set of informative sensory features? We argued that, when performance requires inferences limited by sampling of the statistics of input features, resources should be devoted in proportion to feature variability. A basic idea here is that features that range over a wider range of possible values are less predictable, and will better distinguish between contexts in the face of input noise. We used this hypothesis to successfully predict...”

Reviewer #3

The authors demonstrate that the level of variability in various local, multi-point statistics in natural scenes quantitatively predicts the relative salience of these statistics. I find their data largely convincing, and I feel that this is an important question and that the authors are careful and creative in their approach here. However, I have several concerns, which I suspect in part reflect my own ignorance or misunderstanding about the present paper and past work by these authors.

First, I am not clear on what is qualitatively new here beyond what was shown in the nice 2010 PNAS paper by several of the same authors. I can see that some of the details of their analyses are different, and the comparisons here are more quantitative, but it seems to me that the main ideas and conclusions were already in place in the previous work.

Briefly, the previous work (Tkačik et al., 2010) demonstrated a close relationship between which image statistics are analyzed by the visual system, and the statistics of natural images: resources are devoted to image statistics that cannot be predicted from simpler ones (Tkačik et al., 2010). Here, we build on this, and examine allocation of resources within this identified set of informative images. The previous work made no attempt to predict this allocation, and the current work is not applicable to image statistics that are uninformative (the distinction made in the previous work).

Previous work considered natural scene statistics defined based on local correlations defined over different spatial configurations of pixels. It demonstrated that these statistics can be divided into two groups, informative and uninformative. Informative features – higher-order statistics whose value cannot be deduced from lower-order statistics – are encoded, while uninformative features are not. Indeed, the visual system would be wasteful if it were to invest in mechanisms to encode uninformative statistics, and evidence from Tkačik et al. (2010) suggests that this does not happen. This advance was crucial; as one looks at high-order statistics, there is an exponential explosion of correlations that we (or the brain) could compute. Tkačik et al. (2010) showed that it only makes sense to focus on ones that are informative, and that “informativeness” can be determined from natural scenes alone.

The current manuscript takes this finding as a starting point for exploring the relative allocation of resources to encoding the set of nine informative statistics defined by the 2 × 2 square glider. Unlike Tkačik et al. (2010), which simply identified this set of statistics as informative, here we make and confirm dozens of quantitative predictions concerning how resources are allocated to encode them. The fact that an application of the efficient coding principle is successful in characterizing resource allocation is by no means a straightforward consequence of Tkačik et al. (2010), as any allocation of resources within this parameter set would have been consistent with those previous results.

To clarify this distinction, we have schematized the existence of informative vs uninformative features in a newly-added figure (Figure 4), and in the associated text. We note that the existence of uninformative statistics was the focus of previous work, and we note that the present work focuses on the optimal allocation of resources among those statistics that are informative:

Results: “Successive stages of sensory processing share the same broad goals: invest resources in encoding stimulus features that are sufficiently informative, and suppress less-informative ones. In the periphery, this is exemplified by the well-known suppression of very low spatial frequencies; in cortex, this is exemplified by insensitivity to high-order correlations that are predictable from lower-order ones. Previous work has shown that such higher-order correlations can be separated into two groups – informative and uninformative – and only the informative ones are encoded (Tkačik et al., 2010). We used this finding to select an informative subspace for the present study, and we asked how should resources should be efficiently allocated amongst features within this informative subspace.”

The authors pitch their results in terms of cortical processing, but the only data presented are from human psychophysics, used here as a proxy for cortical processing. The authors give arguments for why salience strongly reflects cortical processing. But I feel that the claims about cortical processing are stated too strongly in some parts of the paper given that they do not directly measure cortical activity nor perform any manipulations to cortex, for example.

We rewrote this section, acknowledging that the inference is indirect and explicitly stating that we did not record cortical responses. But we do feel that the evidence that psychophysical performance reflects cortical processing is compelling, as it is based on independent lines of evidence that, individually, are strong: briefly, stimuli are many times above contrast and resolution thresholds; task performance requires pooling over wide areas; the computations required for the task are more complex than what is considered to occur in the retina; and physiologic recordings show that neural responses to high-order correlations are not present in the thalamus, but are present in visual cortex, and much more so in V2 than in V1.

Specifically, we state:

Results: “Although we did not record cortical responses directly, several lines of evidence indicate that that the perceptual thresholds we measured are determined by cortical processes. […] Conversely, macaque visual cortical neurons (Purpura et al., 1994), especially those in V2, manifest responses to three- and four-point correlations (Yu et al., 2013).”

Because this evidence is admittedly indirect, we added the modifier “likely” to the subheader for this section, to read “Perceptual salience of multi-point correlations likely arises in cortex.”

It seems to me that the dichotomy set up by the authors between predictions for peripheral and central processing is somewhat overstated. For example, in the final sentence of the Abstract, they state that the efficient coding hypothesis applies in “a different guise” and makes “different predictions” for the central nervous system than the periphery. They predict that sensitivity in the CNS should be greater for highly variable stimuli, but it is not clear to me that this is not one of the predictions one would make for peripheral sensory neurons using this principle.

We see how a ‘dichotomy’ is an oversimplification, and a fairer description is that of two qualitatively different regimes (“whitening” and input-noise limiting) depending on the relative strengths of input and output noise.

As mentioned in our response to Reviewer 1, we clarified our language throughout the Discussion to emphasize that the efficient coding regimes are a general feature of information optimization, and neither regime is a priori restricted to peripheral vs central processing. Thus, even in sensory periphery, there might exist special regimes when the signal is input-noise dominated. The best known example is perhaps that of night vision, when photon shot noise is no longer negligible and qualitatively reshapes peripheral information processing. Instead of decorrelation by center-surround receptive fields, receptive fields averaging the signal, i.e., they apply gain that enhances the correlations already present in the input in order to fight the detrimental effects of input noise. These special considerations, which are relevant for night vision, do not apply generally to the extensively-studied area of daylight vision.

We would like to emphasize clearly that the constraints considered here (nonlinear image features subject to sampling noise) thus differ from those considered by some of the most successful applications of efficient coding in the periphery. Because of these constraints, our finding that sensitivity increases with signal variability differs qualitatively from the findings of these studies, where whitening is optimal (or near optimal, see Doi and Lewicki (2014)) and sensitivity decreases with signal variability. Moreover, since feature extraction is thought to be one of the main tasks of central vision, the sampling noise constraint is likely to represent a general aspect of cortical processing, rather than a special case as for night vision in the periphery.

In addition to changes in the text, we have added a fourth figure that illustrates these two regimes in a simple parallel-channel model. As an illustration of the regime in which output noise dominates, we discuss the well-known suppression of low spatial frequencies (Discussion). We then hypothesize that the other regime (input noise dominating) is relevant for central processing (Discussion). We discuss the specific application of efficient coding in this context, which differs from an application in the sensory periphery. Here, we apply efficient coding to complex nonlinear features (counts of glider colorings), where sampling fluctuations in glider counts provide a source of input noise.

We removed language that dichotomizes peripheral vs central processing, such as “it applies in a different guise and makes different predictions” (formerly the final sentence of the Abstract). Instead, we call upon the previous successes of efficient coding in the periphery in order to highlight one particular regime of efficient coding (whitening), and we use these examples to contrast the qualitative findings of our study.

What was the exact task being performed by the subjects?

We added the following clarification about the psychophysical task, and we refer the reader to Methods, Psychophysical methods, for additional details:

Results: “To characterize perceptual sensitivity to different statistics, we isolated them in synthetic visual images and used a figure/ground segmentation task (Figure 2B). We used a four-alternative forced-choice task in which stimuli consisted of a textured target and a binary noise background (or vice-versa). Each stimulus was presented for 120ms and was followed by a noise mask. Subjects were then asked to identify the spatial location (top, bottom, left, or right) of the target. Experiments were carried out for synthetic stimuli in which the target or back- ground was defined by varying image statistic coordinates independently (Figure 2A shows examples of gamuts from which stimuli are built).”

What would Figure 1E look like for IId samples from white (or colored) noise?

When natural images are replaced with white noise, the variation is identical across individual image statistics. (The same is true for colored noise, since our processing pipeline includes whitening.) While this is mathematically guaranteed, it may not be obvious, and we agree that showing it is helpful. We now include a sentence within the main manuscript:

Results: “Interestingly, third-order correlations are the least variable across image patches. An analogous analysis performed on white noise yields a flat distribution with considerably smaller standard deviation values (See Methods, Analysis variants for Penn Natural Image Database, and Figure 1–figure supplement 3 for comparison). These (and subsequent) findings are preserved across different choices of image analysis parameters...”

We have also included an additional Figure in Methods, Analysis variants for Penn Natural Image Database that illustrates this comparison, along with a corresponding addition to the text:

Methods “In Figure 1–figure supplement 3, we show that the relative variation in different image statistics (first shown in Figure 1E) is not an artifact of our image analysis pipeline, as the pattern of variation is destroyed if white-noise image patches are instead used.”

Figure 1–figure supplement 3. Image statistics along single coordinate axes for white-noise patches. The robustly observed statistical structure of natural scenes (open circles) is completely absent from the same analysis performed on samples of white noise (shaded circles). The inset shows that this holds across analysis parameters.

How do the results change for images pixelated along a grid that is 45 degrees tilted relative to the vertical and horizontal directions?

Of course we predict that there would still be a close correspondence between image statistics and psychophysics. But testing this; and doing so in a way that accurately isolates what would likely be a subtle effect of the grid rotation, is unfortunately not practical. Re-photographing the images with an oblique sensor would be required to avoid the artifacts that would arise from digital rotation and resampling. Collecting a parallel set of psychophysical data would require approximately 200,000 additional psychophysical judgments (the better part of a year at a humane pace). We hope the reviewer understands.

Related to this, could the√2 differences in the degree of variation for the vertical/horizontal and diagonal beta data points in Figure 3A be due to the difference in the distance between the points in question given the orientation of the pixelation grid?

It is an interesting observation, but there is more going on than just a difference in center points: for vertical/horizontal two-point sensitivities, the checks involved share a common edge, while for the diagonal case, they only share a corner. Conversely, other experiments show that increasing the spacing between

checks (in either the cardinal or diagonal directions) has very little effect on the sensitivities, over a fivefold range including the range used here (Conte et al., 2014). So we think that the finding that the √ 2 is likely to be a coincidence. We’d therefore prefer not to mention this point, ratio is approximately
as mentioning it but then describing the above evidence would likely be viewed as a distraction.

But we do now emphasize that the diagonal and cardinal directions differ by configuration and not just orientation, so that it is clear that the difference in sensitivities is not simply an “oblique effect.” We have clarified this with the following addition to the main text:

Results: “Note that the difference between the sensitivities in the horizontal and vertical directions (β− and β|) vs the diagonal directions (β\ and β/) is not simply an “oblique effect”, i.e., a greater sensitivity cardinally- vs obliquely-oriented contours (Campbell et al., 1966). Horizontal and vertical pairwise correlations differ from the diagonal pairwise correlations in more than just orientation: checks involved in horizontal and vertical pairwise correlations share an edge, while checks involved in diagonal pairwise correlations only share a corner. Correspondingly, the difference in sensitivities for horizontal and vertical correlations vs diagonal correlations is approximately 50%, which is much larger than the size of the classical oblique effect (10–20%, see Campbell et al. (1966)).”

Author details

1. Ann M Hermundstad

   1. Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, United States
   2. Laboratoire de Physique Théorique, École Normale Supérieure, Paris, France

   Contribution

   AMH, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Contributed unpublished essential data or reagents

   For correspondence

   annherm@physics.upenn.edu

   Competing interests

   The authors declare that no competing interests exist.

2. John J Briguglio

   Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, United States

   Contribution

   JJB, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Contributed unpublished essential data or reagents

   Competing interests

   The authors declare that no competing interests exist.

3. Mary M Conte

   Brain and Mind Research Institute, Weill Cornell Medical College, New York, United States

   Contribution

   MMC, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Contributed unpublished essential data or reagents

   Competing interests

   The authors declare that no competing interests exist.

4. Jonathan D Victor

   Brain and Mind Research Institute, Weill Cornell Medical College, New York, United States

   Contribution

   JDV, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Contributed unpublished essential data or reagents

   Contributed equally with

   Vijay Balasubramanian and Gašper Tkačik

   Competing interests

   The authors declare that no competing interests exist.

5. Vijay Balasubramanian

   1. Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, United States
   2. Laboratoire de Physique Théorique, École Normale Supérieure, Paris, France
   3. Initiative for the Theoretical Sciences, City University of New York Graduate Center, New York, United States

   Contribution

   VB, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Contributed unpublished essential data or reagents

   Contributed equally with

   Jonathan D Victor and Gašper Tkačik

   Competing interests

   The authors declare that no competing interests exist.

6. Gašper Tkačik

   Institute of Science and Technology Austria, Klosterneuburg, Austria

   Contribution

   GT, Conception and design, Acquisition of data, Analysis and interpretation of data, Drafting or revising the article, Contributed unpublished essential data or reagents

   Contributed equally with

   Jonathan D Victor and Vijay Balasubramanian

   Competing interests

   The authors declare that no competing interests exist.

• 2,827

  Page views

• 478

  Downloads

• 42

  Citations

Article citation count generated by polling the highest count across the following sources: Scopus, Crossref, PubMed Central.