Foveated metamers of the early visual system

We constructed foveated models of human perception that capture sensitivity to local luminance and spectral energy (see Figure 2). Both models are ‘pooling models’ (Balas et al., 2009; Freeman and Simoncelli, 2011; Keshvari and Rosenholtz, 2016; Wallis et al., 2019), which compute statistics as weighted averages within overlapping local windows. A specific pooling model is characterized by both the quantities that are pooled and the shapes/sizes of the pooling windows. These regions are analogous to neural receptive fields, whose sizes grow proportionally with distance from the fovea, as documented in monkey physiology and human fMRI (e.g., Gattass et al., 1981; Wandell and Winawer, 2015).

Figure 2

Download asset Open asset

As in previous studies (Freeman and Simoncelli, 2011; Keshvari and Rosenholtz, 2016; Wallis et al., 2019), we used arrays of smooth overlapping windows that are separable and of constant size when expressed in polar angle and log-eccentricity (consistent with the approximate log-polar geometry of visual cortical maps; Schwartz, 1977), as illustrated in Figure 2. Like these previous studies, our windows were radially elongated (the radial extent is roughly twice the angular extent), but unlike the previous studies, we used Gaussian profiles separated by one standard deviation, yielding a smoother representation and minimal ringing and blocking artifacts in the synthesized stimuli (see Appendix 3 for a comparison of the two window profiles). The log-polar mapping ensures that the size of the pooling windows, in both radial and angular directions, is proportional to their distance from the fovea, and this proportionality is controlled by a single scaling parameter, $s$. Increasing this value increases the size of all pooling regions, resulting in a larger and more diverse set of metameric stimuli, in a nested manner (Figure 3).

Figure 3

Download asset Open asset

In the current study, we examine two models that compute different statistics within their pooling windows. The luminance model pools pixel intensities, and thus, two luminance model metamers have the same average luminance within corresponding pooling windows. This model serves as a baseline for comparison to models that pool more image statistics, such as the energy model used here and the texture model used in Freeman and Simoncelli, 2011; Wallis et al., 2019. Since the model’s responses are insensitive to the highest frequencies, luminance model metamers include blurred versions of the target image (in which high frequencies are discarded), but also variants of the target image in which high frequencies are randomized or even amplified. In general, synthesized luminance model metamers inherit the high-frequency content of their initialization image, as can be seen in Figure 4, middle row. While the high-scaling model metamer is clearly perceptually distinct from the target image (regardless of observer fixation location), the low-scaling stimulus is difficult to discriminate from the target when fixating at the center of the stimulus (i.e., when the human observer’s fovea is aligned with the model fovea).

Figure 4

Download asset Open asset

The spectral energy model pools the squared outputs of oriented bandpass filter responses at multiple scales and orientations, as well as the pixel intensities. As such, its statistics are a superset of the luminance model, and its metamers are a subset. The energies are computed using a complex steerable pyramid, which decomposes images into frequency channels selective for six different scales and four different orientations. Energy is computed by squaring and summing the real and imaginary responses (arising from even- and odd-symmetric filters) within each channel. These energies, along with the luminances, are then averaged within the spatial pooling windows. Thus, a pair of energy model metamers have the same average oriented energy and luminance within each of these windows. The bottom row of Figure 4 shows energy model metamers for two different scaling values. The low scaling value for the energy model is approximately matched to the higher scaling value for the luminance model, while the higher scaling value is approximately that of the energy model from Freeman and Simoncelli, 2011. The high-scaling model metamer is perceptually distinct from the target stimulus and also perceptually distinct from the high-scaling luminance model metamer. The low-scaling model metamer, on the other hand, is difficult to distinguish from the original image (when fixating at the center), but is readily distinguished when one fixates peripherally.

The appearance of these two model metamers reflects both the measurements that are being matched and the seed images used to initialize synthesis. The luminance model matches average pixel intensity but has no constraints on spatial frequency, and thus its metamers retain the high-frequency content of the initial white-noise images. The energy model, on the other hand, matches the average contrast energy at all scales and orientations but discards exact position information (which depends on phase structure). Hence, unlike the luminance model metamers, it reduces the high-frequency power to match the typical content of natural images and essentially scrambles the phase spectrum, leading to the cloud-like appearance of its metamers. For both models and all scales tested, the synthesized stimuli are physically very different from the target stimuli, as measured by mean-squared error between the images (see Appendix 5).

As can be seen in Figure 4, both models can generate perceptual metamers. More generally, all pooling models can generate perceptual metamers if the scaling value is made sufficiently small (in the limit as scaling goes to zero, the model metamers must be identical to the target, in every pixel). For statistics that capture features relevant to human perception, metamers can be achieved with windows whose size is matched to the scale of human perceptual sensitivity. The maximal scaling at which synthetic stimuli are perceptual metamers is thus highly dependent on the choice of underlying statistics: in our examples, the energy model perceptual metamer (Figure 4, bottom left) is generated with a scaling value about six times larger than that for the luminance model perceptual metamer (middle left), and about five times smaller than those reported in Freeman and Simoncelli, 2011 using higher-order texture statistics. The goal of the present study is to use psychophysics to find the largest scaling value for which these two models generate perceptual metamers, known as the critical scaling.

We synthesized model metamers matching 20 different natural images (the target images) collected from the authors’ personal collections, as well as from the UPenn Natural Image Database (Tkačik et al., 2011, extended dataset of urban images provided by David Brainard). The images were chosen to span a variety of natural image content types, including buildings, animals, and natural textures (Figure 5). Model metamers were generated via gradient descent on the squared error between target and synthetic pooled statistics, and initialized with either an image of white noise or another image drawn from the set of target images.

Figure 5

Download asset Open asset

In the experiments, observers discriminated two grayscale stimuli, of size 53.6 by 42.2°, sequentially displayed. Each stimulus was separated into two halves by a superimposed vertical bar (mid gray, 2° wide, see Figure 15). One side, selected at random on each trial, was identical in the two intervals, while the other differed (e.g., during the second interval, one half of the stimulus contains the target image, the other a synthesized model metamer). Each stimulus was presented for 200 msec, separated by a 500 msec blank (mid gray) interval, and followed by a blank screen with text prompting the observer to report which side of the stimulus had changed.

There are several reasons to think that the subjects generally maintained good fixation (see section ‘Apparatus’). However, we did not use eye tracking in the psychophysical experiment, and thus cannot guarantee that our observers maintained fixation for all trials.

We fit the behavioral data using the two-parameter function introduced in Freeman and Simoncelli, 2011, estimating the critical scaling ($s_c$) and maximum $d^{\mathrm{\prime }}$ ($\alpha$) parameters with a Markov Chain Monte Carlo (MCMC) procedure and a hierarchical, partial-pooling model similar to that used by Wallis et al., 2019. For a given model and comparison, performance increases monotonically with scaling, and is fit well by this particular psychometric function (Figure 6A). The exception is the synthesized vs. synthesized comparison for the luminance model, for which performance remains poor at all scales (see next section). In the original vs. synthesized cases (for both models), performance is near chance for the smallest tested scaling values and exceeds 90% for the largest. The critical scaling values, as seen in Figure 6B, are approximately 0.016 for the luminance model and 0.06 for the energy model. Thus, as expected, the critical scaling for the model depends on the statistic being pooled: humans are sensitive to changes in local luminance at a much smaller scale than local spectral energy.

Figure 6

Download asset Open asset

For both luminance and energy models, it is generally easier to distinguish an original image stimulus from a synthesized stimulus than to distinguish two synthesized stimuli initialized with different white noise seeds (with same target image and scaling value), as also reported in Wallis et al., 2019 for their pooled texture model. For the luminance model, discrimination of two synthesized stimuli is nearly impossible at all scaling values (note that we only have data for one participant, an author, for this comparison; we did not wish to subject the other participants to this nearly impossible task). For the energy model, discriminating two synthesized stimuli is possible but difficult, with performance only approaching 60%, on average (although note that there are substantial differences across subjects, see Figure 6B and Appendix 6). The critical scaling value for this comparison, approximately 0.25, is comparable to that reported in Freeman and Simoncelli, 2011 for their pooled energy model. The asymptotic performance, however, is much lower in our data. We attribute this to experimental differences (see Appendix 4).

The difficulty of differentiating between two synthesized stimuli is striking, as illustrated in Figure 7. In the limit of global pooling windows, luminance metamers are samples of white noise, which cannot be distinguished when presented at the resolution and extent used for the stimuli in this study (Wallis et al., 2019 made a similar point when discussing their use of the original vs. synthesized task). Analogously, synthesis with the energy model forces local oriented spectral energy to match, without explicitly constraining the phase. Two instances of phase scrambling within peripheral windows are not easily discriminable, even though either of the two might be discriminable from a stimulus with more structure. The difficulty discriminating between stimulus pairs like those shown in Figure 7 may arise early or late in processing, up to and including memory. Wherever this difficulty arises, it is more pronounced for the synth vs. synth comparisons. These results complicate the interpretation of the critical scaling value, which we return to in the discussion.

Figure 7

Download asset Open asset

To the extent that the models (at critical scaling) capture something important about human perception, stimulus pairs that are model metamers will be perceptual metamers, and hence discrimination should be at chance. Neither model offers predictions of perceptual discriminability (they are deterministic, and do not specify any method of decoding or comparing stimuli). Consistent with this, the critical scaling, which measures the point at which stimulus pairs become indistinguishable, does not vary much across target images for a given model and comparison, unlike performance at super-threshold scaling values and the asymptotic levels of $d^{\mathrm{\prime }}$ (Figure 6B). Variations in max $d^{\mathrm{\prime }}$ are especially clear in the target image-specific psychometric functions for the original vs. synthesized energy model comparison (Figure 8). Specifically, for the llama target image, performance only rises slightly above chance, even at very large scaling windows. The respective target images in panel Figure 8B suggest an explanation: much of the llama image is cloud-like, while the nyc image is full of sharp edges in the cardinal directions, which arise from precise alignment of phases across positions and scales. As discussed above, synthetic energy model metamers have matching local oriented spectral energy, with randomized phase information; in order to generate sharp, elongated contours for the buildings of nyc, the windows must be very small. Conversely, the appearance of the llama is captured even when the pooling windows are large. Thus, when scaling is larger than critical scaling, some comparisons become easy and some do not. However, this pattern depends on the interaction between the model’s sensitivities and the target image content; this pattern does not hold for the luminance model, or for synthesized vs. synthesized comparisons, for which both the llama and nyc target images exhibit typical performance. See Appendix 6 for more details.

Figure 8

Download asset Open asset

We thus find evidence that although super-threshold performance depends on the interaction between target image content and model sensitivities, critical scaling is consistent across target images, in contrast with Wallis et al., 2019.

The two types of comparisons shown in Figure 6 — original vs. synthesized and synthesized vs. synthesized—show very different critical scaling values. Specifically, for a particular scaling value and set of image statistics, some stimulus pairs are much easier to discriminate than others. We hypothesize that metamers synthesized from white noise seeds are restricted to a relatively small region of the full set of model metamers. As a result, these stimuli are more perceptually similar to each other than they are to the target stimulus. To generate metamers outside of this set, we also used other natural images from our data set to initialize the synthesis procedure (which was not done in previous studies, Freeman and Simoncelli, 2011; Wallis et al., 2019).

Figure 9A shows behavior for these additional comparisons in a single subject, sub-00. Changing the initialization image has a large effect on the synthesized vs. synthesized comparison but little-to-no effect on the original vs. synthesized comparison. For synthesized vs. synthesized, initializing with a different natural image improves performance compared to initializing with white noise, but is still worse than performance for original vs. synthesized.

Figure 9

Download asset Open asset

Figure 9B shows three comparisons involving five metamers arising from different initializations, each with scaling corresponding to the vertical line in panel Figure 9A, but with dramatically different human performance. The top row shows the easiest comparison, between the original image stimulus and a synthesized stimulus initialized with a different natural image (bike); the subject was able to distinguish these two stimuli with near-perfect accuracy (in this case, when comparing against a natural image stimulus, performance is identical regardless of whether the metamer was initialized with white noise or natural image). The bottom row shows the hardest comparison, between two synthesized stimuli initialized with different samples of white noise. As discussed above, comparing two stimuli of this type is difficult even with large pooling windows; at this scaling level, humans are insensitive to the differences between them, and so performance was at chance. The middle row shows two synthesized stimuli, initialized with different natural images, which the subject was able to distinguish with moderate accuracy. When comparing these two stimuli, one can see features in the periphery that remain from the initial image (e.g., traces of the lines between tiles are present in the bottom-right corner of the tile-initialized stimulus, while remnants of the power lines are present in the top right of the highway-initialized stimulus). Even when fixating, the subject was able to use these features to distinguish the two stimuli, that is, the human was sensitive to them while the model was not. This reinforces the notion that the initialization of the synthesis process is important. In both the middle and bottom row, both stimuli are synthesized (i.e., neither row contains the target image), yet one comparison is much harder than the other (see also the critical scaling values for these comparisons in Figure 10).

Figure 10

Download asset Open asset

Note that the experiments discussed in this section were only carried out by a single participant. We do not believe this substantially affects our conclusions. This subject was an author (and thus an expert observer, in that they were very familiar with the stimuli) and has an average critical scaling and the highest max $d^{\mathrm{\prime }}$ value for the comparisons we have multiple subjects’ data (see Figure 6); as we largely interpret the critical scaling, values of which are quite consistent across observers, their results are therefore likely typical. While our ability to precisely estimate this critical scaling is limited by the number of subjects, we believe that the broad trends that we focus on are likely robust. That is, while the exact critical scaling values will likely vary across participants, we believe that starting from a natural image will have a negligible effect on the critical scaling of the energy model original vs. synth task, while doing so for the energy model synth vs. synth task will lead to a result intermediate between the two energy model white-noise initialization comparisons. Finally, we gathered 3 hours of data for these two energy model natural image comparisons, allowing us to fit the full psychophysical curve. See the section ‘Session and block organization’ for a full enumeration of the trials.

As with the comparison (but unlike target image content), the initialization of the synthesis procedure has a substantial effect on our experimental estimates of critical scaling.

For each model, the number of statistics is proportional to the number of pooling regions and thus decreases quadratically with scaling. Table 1 shows average critical scaling values across all conditions, along with the corresponding number of model statistics. We can see that critical scaling does not correspond to a fixed number of statistics. We should also note that if one were to use the model outputs as a compressed representation of the image, the number of statistics in each representation is almost certainly an overcount for several reasons. First, in order to ensure that the Gaussian pooling windows uniformly tile the image, the most peripheral windows in the model have the majority of their mass off the image, which is necessary to avoid synthesis artifacts. Second, for the energy model, we did not attempt to determine how the precise number of scales or orientations affected metamer synthesis, and currently all scales are equally weighted across the image. As the human visual system is insensitive to high frequencies in the periphery and low frequencies in the fovea, this is probably unnecessary, and so some of these statistics can likely be discarded. Finally, our pooling windows are highly overlapping and thus the pooled statistics are far from independent; this redundancy means that the effective dimensionality of our model representations is less than the quoted number of statistics.

In summary, critical scaling does not reflect a fixed number of statistics. We instead believe it reflects the spatial scale at which human perception is sensitive to changes in each of these statistics.

Visual representations are formed through a cascade of transformations. An appealing hypothesis is that each of these is comprised of the same canonical ‘blurred feature extraction’ computation, differing only in the choice of feature and the spatial extent of the blurring (Fukushima, 1980; Douglas et al., 1989; LeCun et al., 1989; Heeger et al., 1996; Riesenhuber and Poggio, 1999; Bruna and Mallat, 2013). In the perception literature, Lettvin, 1976 provides an early, informal description of this ‘compulsory feature averaging’ as an explanation for the degradation of peripheral percepts, non-foveated versions have been described in Parkes et al., 2001; Pelli et al., 2004; Greenwood et al., 2009, and foveated proposals appear in Balas et al., 2009; Freeman and Simoncelli, 2011. In these, stages are distinguished by their features and the scaling of the pooling regions with eccentricity. Similarly, in the primate visual system, features represented in successive stages of processing become more complex and receptive fields increase in size. In particular, the optics and photoreceptors pool the incident light over small regions, V1 pools spectral energy over larger regions, V2 pools texture-related statistics over yet larger regions, and so forth. Consistent with this view, we find that for synthetic metamer stimuli, the features pooled by the model have a dramatic effect on the critical scaling value, which is approximately four times larger for the energy model than for the luminance model in the original vs. synthesized comparison (Figure 6 and Table 1). The values of critical scaling as a function of image statistic and comparison type are summarized in Figure 11, along with the results of Freeman and Simoncelli, 2011 and Wallis et al., 2019 obtained for a model of texture (Portilla and Simoncelli, 2000).

Figure 11

Download asset Open asset

Overall, critical scaling for original vs. synthetic comparisons follows approximate ratios of 1:4:12 for luminance:spectral-energy:texture features (filled circles, Figure 11).

We found large effects of comparison type on performance (Figure 11, filled vs. unfilled). Specifically, for the energy model, the critical scaling for synthesized vs. synthesized was about four times larger than that for original vs. synthesized. For the luminance model, participants were generally unable to discriminate any pairs for the synthesized vs. synthesized comparison. These effects arise from an interaction between human perception, the model, and the synthesis process.

As shown in Figure 12, the synthesis procedure can bias the sampling of model metamers such that two synthesized stimuli are metameric with each other but distinguishable from the target stimulus. The idealized version of the metamer paradigm implicitly assumes that the synthesized stimuli sample the manifold of possible model metamers broadly, but our synthesis procedure (as with that of Freeman and Simoncelli, 2011 and Wallis et al., 2019) does not guarantee this.

Figure 12

Download asset Open asset

The conceptual framework we outlined in this paper emphasizes deterministic information loss, arising from projections of high-dimensional visual images onto a lower-dimensional manifold. Under these conditions, metamerism is transitive: all stimuli within an equivalence class are indistinguishable from each other, and all stimulus pairs from different classes are potentially distinguishable. However, reductions in discriminability can also arise from stochastic information loss: discrimination thresholds are governed by noisy neural responses, as developed in the frameworks of signal detection theory and ideal observer models. The psychophysical function we fit to the data arises from an observer model that compares local statistics of stimuli (Freeman and Simoncelli, 2011) and combines the two forms of information loss. It asserts that performance falls exactly to chance when the scaling used for synthesis is less than or equal to the true scaling (i.e., when the stimuli are metameric, projecting to the identical point in perceptual space). As the synthesis scaling exceeds the true scaling, the measured differences in stimulus statistics exceed the level of noise in subjects’ perceptual systems, and performance gradually improves. Our analysis thus focuses on the critical scaling, the point at which the piecewise function first rises above chance, which is the primary prediction of the metamer paradigm. In practice, our metamer models, like all models (Box, 1976), are imperfect, but small errors in the models are likely masked by subjects’ stochastic noise during discrimination.

The effect of comparison type was previously reported by Wallis et al., 2019 and Deza et al., 2019 in models pooling texture-like statistics. As seen in Figure 11, the ratio in critical scaling between the two types of comparisons decreases as the model features become more complex: infinite for luminance, roughly quadruple for energy, and less than double for texture. One potential explanation for this observation is that these computations are being performed in deeper stages of the visual hierarchy, and there are progressively fewer opportunities to discard information later in the hierarchy. For example, the difference in V1 responses for a pair of stimuli may be discarded in later stages, whereas in a later area (e.g., area IT), there are fewer remaining stages of processing in which information can be discarded. This may explain why we see no overlap between the critical scaling values for original vs. synthesized and synthesized vs. synthesized comparisons across target images in Figure 6B, whereas Wallis et al., 2019 find substantial overlap for the texture model.

Another potential explanation is schematized in Figure 13: for the luminance model, the model metamer classes grow with scaling $s$ in such a way that model metamers generated from white noise initialization fall within the same perceptual metamer class for a wide range of $s$ values. This is an extreme case of the synthesis issue described in the previous section. Conversely, as the texture model’s critical scaling dependence on comparison type is much weaker, we can hypothesize that its metamer classes are such that model metamers synthesized from white noise readily fall into distinct perceptual metamer classes, leading to a critical scaling value that is similar to the value for the original vs. synth comparison.

Figure 13

Download asset Open asset

Our experimental results demonstrate that critical scaling depends on the comparison type and feature complexity. We also find that it depends on synthesis initialization. Specifically, the set of metamer stimuli generated for our experiments is a biased sample from the space of all possible model metamers, and is particularly dependent on the distribution of initialization images. This effect is especially noticeable in the case of the luminance model, as discussed in the previous section. We conjectured that natural images are more likely than synthetic images to include information that the human visual system has evolved to discriminate and hence doesn’t discard even in later processing. As such, initialization with natural images can provide an intuitive method of sampling from a different portion of the manifold of possible model metamers. The schematic presented in Figure 14 provides an illustration of how this might occur.

Figure 14

Download asset Open asset

We controlled the generation of metamer stimuli in our experiments by selecting one of two initialization distributions: white noise and natural images. A natural intermediate choice would be pink noise, with frequency spectrum matched to that of natural images. Such initialization would reduce the high frequencies found in the super-threshold luminance model metamers (e.g., Figure 4), while avoiding the detectable features found in those initialized with natural images (see Appendix 2 for illustrative example images). This would result in another psychophysical curve on Figure 9A, shifted to the right of the leftmost (original vs. synth: white noise) curve for this model, and thus a higher estimated critical scaling value. The logic of our experiments, however, relies on finding the smallest critical scaling value across initializations, rather than generating stimuli that look as similar to the target as possible. This reiterates the importance of thinking carefully about synthesis initialization for metameric stimulus generation, but we expect such initializations would not otherwise influence our interpretation of the results. A more principled statistical sampling approach could result in metamers that better reflect natural image properties, or metamers that are more likely to be discriminable by human observers.

Similar to Wallis et al., 2019 and Brown et al., 2023, we find that metamer discrimination performance is somewhat dependent on image content. Both of those studies synthesize model metamers based on pooled texture statistics, and Wallis et al., 2019 show that texture-like target image stimuli are harder to distinguish from their synthesized stimuli than scene-like ones, while Brown et al., 2023 show that original textures with higher global and local regularity (e.g., woven baskets) are easier to distinguish from their synthesized stimuli than those with low regularity (e.g., animal fur). This aligns with our result: the most distinguishable pairs include natural image stimuli features not well-captured by the synthesizing model, whereas the least distinguishable include those natural image stimuli whose features are all adequately captured.

However, we should note that we found this image-level variability largely in super-threshold performance, and this variability does not constitute a failure of these pooling models. As pointed out by Freeman and Simoncelli, 2011, asymptotic performance also varies with experimental manipulation, while critical scaling remains relatively unaffected. The metamer paradigm makes strong predictions about what happens when the representations of two stimuli are matched: they are indistinguishable, and so performance on a discrimination task will be at chance, as captured by the critical scaling value. However, it makes no predictions about performance at super-threshold levels. An analogy with color vision seems apt: color matching experiments provide evidence for what spectral distributions of light are perceived as identical colors, but provide no information about whether humans consider two shades of blue distinguishable or blue more similar to green or to red; further measurements are necessary to understand color appearance.

By investigating how such differences are handled by later brain areas, this complementary approach could gain a better understanding of stimulus discriminability beyond ‘identical or not’. Such work could use the models presented here as a starting point and could draw on the substantial literature of observer models in vision science and image processing; see Ziemba and Simoncelli, 2021; Kurzawski et al., 2024 for examples applying this approach to pooling model metamer images. Combining and extending the synthesis-focused metamer approach with observer models’ attention to super-threshold performance would help lead to a fuller understanding of human perceptual sensitivities and insensitivities.

In summary, we’ve used large field-of-view stimuli matched to the foveation properties of the human visual system in order to estimate the spatial precision of the perceptual representation of low-level image features. By improving our understanding of the information that is discarded, these measurements provide a starting point for building a foveated model of the visual system’s front end. We’ve also provided a unified explanation of the variations in critical scaling that arise from changes in image content, comparison type, and synthesis initialization, providing guidance on how to use model and perceptual metamers for understanding visual representations.

We synthesized model metamers matching 20 different natural images (the target images) from the authors’ personal collections, as well as from the UPenn Natural Image Database (Tkačik et al., 2011, extended dataset of urban images provided by David Brainard). The selected photos were high-resolution with 16-bit pixel intensities proportional to luminance, that had not undergone lossy compression (which could result in artifacts). They were converted to grayscale using scikit-image’s color.rgb2gray function (Walt et al., 2014), cropped to 2048 by 2600 pixels (the Brainard photos were 2014 pixels tall, so a small amount of reflection padding was used to reach 2048 pixels), and had their pixel values rescaled to lie between 0.05 and 0.95. Synthesized stimuli were still allowed to have pixel values between 0 and 1; without rescaling the target images, synthesis resulted in strange artifacts with pixels near 0, as this was the minimum allowed value. The target images were chosen to span a variety of natural image content types, including buildings, animals, and natural textures (see Figure 5).

We synthesized the model metamers using custom software written in PyTorch (Paszke et al., 2019), using the AMSGrad variant of the Adam optimization algorithm (Kingma and Ba, 2014; Reddi et al., 2018), with learning rate 0.01. Slightly different approaches were used for the luminance and energy model metamers. For the luminance model metamers, the objective function was to minimize the mean-squared error between the model representation of the target and synthesized stimuli, $L(x,\hat{x})=(M(x)-M(\hat{x}){)}^2$, and synthesis was run for 5000 iterations. For the energy model metamers, the objective function also contained a quadratic range penalty term, which penalized any pixel values outside of [0,1], $L(x,\hat{x})=0.5(M(x)-M(\hat{x}){)}^2+0.5\mathcal{B}(\hat{x})$, where

Synthesis was run for 15,000 iterations. Additionally, energy model metamer synthesis used stochastic weight averaging (Izmailov et al., 2018), which helped avoid local optima by averaging over pixel values as synthesis neared convergence, and used coarse-to-fine optimization (Portilla and Simoncelli, 2000). Additionally, each statistic (in both models) was z-scored using the average statistic value computed across the entire image on a selection of grayscale texture images. For both models, synthesis terminated early if the loss had not decreased by more than 10⁻⁹ over the past 50 iterations. While not all model metamers achieved the same loss values, with differences in synthesis loss across target images, there was no relationship between the remaining loss and behavioral performance.

For each model, its windows were represented as two tensors, one for angular slices and one for annuli, which, when multiplied together, would give the individual windows, with separate sets of windows for each scale in the energy model. This required a large amount of memory, and so for scaling values below 0.09, models were too large to perform synthesis on the available NVIDIA GPUs with 32 GB of memory. Thus, all luminance model metamers were computed on the CPU, and synthesis of a single stimulus took from about an hour for scaling 1.5 to 2 days for scaling 0.058 to 14 days for scaling 0.01. For the energy model metamers, the lowest two scaling values were computed on the CPU, with synthesis taking about a week. For those energy model metamers which were able to be computed on the GPU, synthesis took from 5 hours for scaling 0.095 to 1.5 hours for scaling 0.27 and above. This synthesis procedure was completed in parallel using the high-performance computing cluster at the Flatiron Institute.

Synthesized stimuli for original vs. synthesized and synthesized vs. synthesized white noise comparisons (see section ‘Psychophysical experiment’) were initialized with full-field patches of white noise (each pixel sampled from a uniform distribution between 0 and 1). For each model, scaling value, and target image, three different initialization seeds were used. A unique set of three seeds was used for each scaling value and target image, except for the following, which all used seeds {0, 1, 2}:

• Luminance model: azulejos, bike, graffiti, llama, terraces, tiles; scaling 0.01, 0.013, 0.017, 0.021, 0.027, 0.035, 0.045, 0.058, 0.075, and 0.5.

• Energy model: azulejos, bike, graffiti, llama, terraces, tiles; scaling 0.095, 0.12, 0.14, 0.18, 0.22, 0.27, 0.33, 0.4, and 0.5.

For original vs. synthesized and synthesized vs. synthesized natural image comparison, synthesized stimuli for each model, scaling value, and target image were initialized with three random choices from among the rest of the target images.

Pooling model windows are laid out in a log-polar grid, with peaks spaced one standard deviation apart, such that adjacent window functions cross at a value of 0.352 (relative to max of 0.4). They have a single parameter, scaling, which specifies the ratio of the pooling window diameter at full-width half-max in the radial direction, and the distance of its center from the fovea, both in degrees. For example, the pooling windows of a model with a scaling factor of 0.1 have a radial diameter of 1° at 10° eccentricity, 2 at 20°, etc.

Stimulus pixels within 0.5° from the fixation point were exactly matched in our synthesized stimuli, approximating the fovea, where no pooling occurs. Additionally, for small scaling values, windows for some distance beyond this region would be smaller than a pixel, and so the only solution is to match the pixel values in that region directly. For example, with an image resolution of 2048 by 2600 and a display size of 53.6 by 42.2°, models with a scaling value of 0.063 have windows whose diameter at FWHM is smaller than a pixel out to 0.52°, with this number increasing quadratically as scaling decreases, reaching 3.29° for scaling 0.01 (see Appendix 5 for how mean-squared error varied across eccentricity).

Eight people (five women and three men, aged 24–33), including an author (WB), participated in the study and were recruited from New York University. All subjects were graduate students or postdocs in vision science labs who had experience participating in other visual psychophysics experiments, and who were accustomed to performing peripheral tasks while maintaining central fixation. All had normal or corrected-to-normal vision. Each subject completed nine 1-hour sessions. One subject (sub-00, author WB) also performed seven additional sessions. All subjects provided informed consent before participating in the study. The experiment was conducted in accordance with the Declaration of Helsinki and was approved by the New York University ethics committee on activities involving human subjects.

A psychophysical experiment was run in order to determine which of the synthesized model metamers were also perceptual metamers. We first describe the structure of a single trial, then how the trials were organized into blocks and sessions.

Trial structure

Request a detailed protocol

See Figure 15 for schematic. Observers viewed a series of grayscale image stimuli on a monitor, at a size of 53.6 by 42.2°. An initial stimulus, divided in half by a vertical midgray bar 2° wide, was displayed for 200 msec, before being replaced by a midgray screen for 500 msec, followed by a second stimulus for another 200 msec (also divided by a vertical midgray bar). Stimuli were presented for 200 msec to minimize the possibility of eye movements. The dividing bar prevented participants’ use of discontinuities between the two stimulus halves to perform the task. One side of the second stimulus (left half or right half) was identical to the first stimulus, and the other side changed. After the second stimulus was viewed, a midgray screen appeared with text prompting a response, and the observer’s task was to report which half had changed. The observer had as much time as necessary to respond. The two compared stimuli were either two synthesized stimuli (synthesized for identical models with the same scaling value and target image, but different initializations) or one synthesized stimulus and its target image stimulus. Either stimulus could be presented first.

Figure 15

Download asset Open asset

The midgray blank screen presented between the two stimulus presentations reduces motion cues participants could use to discriminate the two stimuli. Our models aim to capture the steady state response to the stimuli, not the transient response. The mask forces the participants to use the image content to discriminate between the two stimuli, rather than relying on temporal edges (analogous to our use of the vertical bar to prevent the use of spatial edges). This introduces a short-term memory component in the task (participants must remember the first stimulus in order to compare it to the second stimulus), as in previous metamer discrimination experiments (Freeman and Simoncelli, 2011; Deza et al., 2019; Wallis et al., 2019). We believe the precise duration of this mask is unimportant for our results: first, Bennett and Cortese, 1996 found the duration of a blank screen did not affect thresholds in a spatial frequency discrimination task over a range from 200 to 10,000 msec, and second, mask duration is likely to have a similar effect on performance as stimulus presentation duration, which Freeman and Simoncelli, 2011 found affected asymptotic performance but not critical scaling.

The stimuli were displayed on an Eizo CS2740 LED flat monitor running at 60 Hz with a resolution of 3840 by 2160 pixels. The monitor was gamma-corrected to yield a linear relationship between luminance and pixel value. The maximum, minimum, and mean luminances were 147.73, 0.3939, and 77.31 cd/m², respectively.

The experiment was run with a viewing distance of 40 cm, giving 48.5 pixels per degree of visual angle. A chin and forehead rest was used to maintain head position, but the subjects’ eyes were not tracked.

The experiment was run using custom code written in Python 3.7.0 using PsychoPy 3.1.5 (Peirce et al., 2019), run on an Ubuntu 20.04 LTS desktop. A button box was used to record the psychophysical response data. All stimuli were presented as 8-bit grayscale images.

A limitation of the study is that we did not track gaze direction. However, there are several reasons to think that fixation breaks do not strongly influence our conclusions. First, the 2° wide vertical bar at the center of the screen ensures that minor deviations in gaze direction from the fixation point would have little effect. Second, each stimulus was only presented for 200 msec, which is the typical latency of saccades (Carpenter, 1988). This does not rule out saccades during the trials, but participants did not have the time to freely explore the images. In principle, participants could have begun the trials gazing away from the fixation point. However, all of our participants were trained observers with experience doing peripheral tasks while maintaining central fixation. Moreover, variation in fixation across observers would likely reveal itself in variability in estimated psychometric curves. But the consistency of estimated parameters (indicated both by their relatively narrow credible intervals and similarity to results reported in Freeman and Simoncelli, 2011) suggests that any such variation did not greatly affect our results. A participant could cheat on the task by, for example, consistently fixating in the corner of the screen. However, such a subject would be an outlier, which we do not see in our data. Indeed, the participant who is an author, and was thus highly motivated to maintain fixation, did not have worse performance than the other observers. Finally, our interpretation of our results relies on the ordering of the critical scaling across models and comparisons, rather than their precise values. Given the consistency of these estimates and their separation from each other, we do not believe the lack of eye tracking affects the overall pattern of results, and hence our conclusions.

All trials were analyzed, a total of 4320 trials per subject per model per comparison (across 15 target images and 8 scaling values) for all energy model comparisons and for luminance model original vs. synthesized white noise comparison. Luminance model synthesized vs. synthesized, white noise comparison had 1440 trials (across five target images and eight scaling values) for a single subject. Where behavioral data is plotted in this paper, the proportion correct is the average across all relevant trials.

We fit psychophysical curves describing proportion correct as a function of model scaling using the two-parameter function for discriminability $d^{\mathrm{\prime }}$ derived in Freeman and Simoncelli, 2011:

where $s_c$ is the critical scaling value (performance is at chance for scaling values at or below $s_c$) and $\alpha$ is the max $d^{\mathrm{\prime }}$ value (called the ‘proportionality factor’ in Freeman and Simoncelli, 2011).

Psychophysical curves were constructed by converting this $d^{\mathrm{\prime }}$ into the probability correct using the same function as in Freeman and Simoncelli, 2011:

where $\mathrm{\Phi }$ is the cumulative of the normal distribution. The probability correct is 50% when $d^{\mathrm{\prime }}=0$ (and thus when scaling is at or below the critical scaling), reaches about 79% when $d^{\mathrm{\prime }}=2$ and 98% when $d^{\mathrm{\prime }}=4$. As the α parameter above gives the maximum $d^{\mathrm{\prime }}$ value, it has a monotonic relationship with the asymptotic performance, which can be seen in Figure 16.

Figure 16

Download asset Open asset

The posterior distribution over parameters $s_c$ and α was estimated using a hierarchical, partial-pooling model, with independent subject- and target image-level effects for both $s_c$ and α, with each model and comparison estimated separately, following the procedure used in Wallis et al., 2019. Subject responses were modeled as samples from a Bernoulli distribution with probability $(1-\pi )P(s)+0.5\pi$, where $\pi$ is the lapse rate, estimated independently for each subject. Estimates were obtained using a MCMC procedure written in Python 3.7.10 (Van Rossum and Drake, 2009) using the numpyro package, version 0.8.0 (Bingham et al., 2018; Phan et al., 2019). MCMC sampling was conducted using the NoU-Turn Sampler algorithm (Hoffman and Gelman, 2014) with parameters selected to ensure convergence, which was assessed using the $\hat{R}$ statistic (Brooks and Gelman, 1998, looking for $\hat{R}$ < 1.01, Vehtari et al., 2021) and by examining traceplots. Parameters were given weakly informative priors and both S_c and α were estimated on natural logarithmic scales.

In sum, for model $m\in \{E,L\}$, comparison $t$, subject $x$, target image $i$, and scaling $s$:

with the following priors:

The priors for $s_{c,mt}$ of the energy and luminance models correspond to critical scales of 0.25 and 0.018, respectively. These values were taken from the centers of the V1 physiological range provided in Freeman and Simoncelli, 2011; Figure 5 and from the slope of a line fit to the dendritic field diameter vs. eccentricity of midget retinal ganglion cells in Dacey and Petersen, 1992; Figure 2B. These priors reflect our original prediction that the models’ critical scaling values should be similar to those of the physiological scaling in the brain area sensitive to the same image features. While we are not committed to this link to physiology, it is reasonable to expect the energy model’s critical scaling to be similar to that of Freeman and Simoncelli, 2011, who tested the same model, and to assume the critical scaling for the luminance model to be a good deal smaller. The priors also reflect our prediction that critical scaling should be independent of comparison type and consistent across target images and subjects, while not placing too much of a constraint on the parameters (see Appendix 7 for a comparison with an unpooled MCMC model, where parameters are estimated independently for each psychophysical curve, and a partial-pooling model that adds interaction terms between subjects and target images).

The posterior distribution represents the model’s beliefs about the parameters given the priors and data and is summarized throughout this paper as the posterior mean and 95% highest density intervals. The latter represents the range of values containing 95% of the distribution with the highest probability, as opposed to the more common 95% confidence interval, which is symmetrically arranged around the mean. The two are identical for symmetric distributions, but can diverge markedly if the distribution is highly skewed (Kruschke, 2015).

These experiments relied on a variety of custom scripts written in Python 3.7.10 (Van Rossum and Drake, 2009), all found in the GitHub repository associated with this paper. The following packages were used: snakemake (Mölder et al., 2021), JAX (Bradbury et al., 2018), matplotlib (Hunter, 2007), psychopy (Peirce et al., 2019), scipy (Virtanen et al., 2020), scikit-image (Walt et al., 2014), pytorch (Paszke et al., 2019), arviz (Kumar et al., 2019), numpyro (Bingham et al., 2018; Phan et al., 2019), pandas (McKinney, 2010; Reback et al., 2021), seaborn (Waskom, 2021), jupyterlab (Kluyver, 2016), and xarray (Hoyer and Hamman, 2017).

Appendix 1—figure 1

Download asset Open asset

Owing to the length of time required to synthesize luminance model metamers with scaling values in the range required for the original vs. synthesized comparison (from 2 days for scaling 0.058–14 days for scaling 0.01), we did not generate a full set of luminance model metamers initialized with natural images and thus did not run the corresponding psychophysical experiment. However, we believe we would obtain similar results as those we observed for the energy model: the performance on the original vs. synthesized comparison would be approximately the same, while it would be greatly improved for the synthesized vs. synthesized comparison, with a smaller critical scaling value and a higher max $d^{\mathrm{\prime }}$. Some example comparisons are presented in Appendix 1—figure 1 to illustrate this. All presented model metamers were generated with scaling 0.23. The bottom row contains two model metamers initialized with different patches of white noise, while the other three model metamers were initialized with other natural images. We only have data for the comparison in the bottom row, where performance was at chance, but, examining the other stimuli, it is likely that performance would be at ceiling for those comparisons. For all model metamers, the pooling windows have grown to the point that a good deal of information from the initial images remains in the ultimate metamer. If these initial images were patches of white noise, that information is impossible for humans to use for discriminating between the two stimuli, but this is not the case if they were natural images. The luminance model predicts all three comparisons would be equally difficult, but the bottom comparison is impossible, while the other two appear to be trivial (based on examining luminance model metamers initialized with natural images at other scaling values, we believe this is true for the range of scaling values tested for the synthesized vs. synthesized comparison, but would need an experiment to understand scaling values in the range tested for the original vs. synthesized comparison).

Appendix 2—figure 1

Download asset Open asset

As discussed in the main text, participants make use of the high frequencies present in the luminance model metamers initialized with white noise when distinguishing them from the target images. One might therefore wonder what luminance model metamers initialized with pink noise, which have the same $1/f$ power distribution found in natural images, would look like. As can be seen in Appendix 2—figure 1, metamers initialized with pink noise have far less power in the high frequencies compared to those initialized with white noise at the same scaling value. Fixating on the center of these images, the white noise-initialized metamer is more discriminable from the target image than that initialized with pink noise. If a full experiment were run, we believe this comparison would lead to a right-shifted psychophysical curve and a higher critical scaling value. However, the logic of our experiments relies on finding the smallest critical scaling value. Thus, while this example shows the importance of synthesis initialization in metamer synthesis, we do not believe it would affect our interpretation of our results or our conclusions.

Previous studies computing the critical scaling of pooling models using psychophysical experiments and model metamers have used the raised-cosine windows described in Freeman and Simoncelli, 2011; Deza et al., 2019; Wallis et al., 2019. In this study, we use Gaussians instead. Though they match the previously used windows in their polar arrangement and radial elongation, one might worry that some difference in their construction might influence our results. The Gaussian windows do overlap more substantially than the raised-cosine windows and, being Gaussians, extend infinitely (unlike the raised-cosine windows), leading to a smoother representation, which resulted in higher-quality synthesized stimuli. Use of these models and the scientific inferences made by the authors do not rely on the exact specification of the windows, and we believe our models can be viewed as comparable to previous results (similarly, Deza et al., 2019 construct a texture-like representation using the representation of VGG19, rather than the Portilla–Simoncelli texture statistics [Portilla and Simoncelli, 2000] used by Freeman and Simoncelli, 2011; Wallis et al., 2019, but their psychophysical results are comparable).

Appendix 3—figure 1

Download asset Open asset

However, interpretation of these results does critically depend on the definition of the model’s scaling parameter, which gives the ratio between the pooling windows’ radial full-width at half-maximum and central eccentricity. We preserve that definition, as can be viewed in Appendix 3—figure 1, which shows an example window from Freeman and Simoncelli, 2011 (constructed using their MATLAB code found on GitHub, Freeman, 2016) and our Gaussian implementation at two scaling values. Both sets of windows were constructed to have a resolution of 512 by 512 pixels, with a maximum eccentricity of 13 and multiple scales; the presented windows are from the second-coarsest scale (thus, a resolution of 256 by 256 pixels) to demonstrate we handled the multi-scale construction similarly as well (similar results are found for other scales).

The left column shows the windows from Freeman and Simoncelli, 2011, the center column shows the windows from this paper, and the right column shows their difference. The windows have been normalized so as they both have a maximum value of 1 (when used in our models, the Gaussian windows have a lower maximum value so that the sum across all windows is 1). Several aspects are visible:

• The centers of the windows are not perfectly aligned. The exact sampling of the two sets of windows is slightly different, which should have no impact on their interpretation.

• The visible extent of the Gaussian is slightly larger. As discussed above, Gaussians extend infinitely, whereas the raised-cosine windows have a finite support. This leads to the desired smoother representation.

• However, the full-width at half-maximum (in both directions) appears to be matched. Scaling therefore has the same interpretation for both sets of windows.

As noted in the text, while the critical scaling we found for the energy model comparing two synthesized stimuli initialized with white noise was comparable to that found in Freeman and Simoncelli, 2011, subjects’ asymptotic performance was lower (average of 60% correct across subjects and target images, compared to 85%). This was also noted, for the ‘mid-ventral’, pooled texture-statistic model, in Wallis et al., 2019, who ran a control experiment to ensure it doesn’t come down to task differences. As our implementation of the pooling models is completely separate from that of Freeman and Simoncelli, 2011, we investigated whether differences in window construction or other implementation details could have led to any marked difference. A Jupyter notebook investigating this can be found in the GitHub repo associated with this project, showing that the windows appear comparable, as do energy model metamers with comparable scaling values. Additionally, if our windows were significantly different from those in Freeman and Simoncelli, 2011, we would expect them to affect the critical scaling, whose value relates window radial diameter to eccentricity. However, the critical scaling value for the only comparison present in both studies (energy model, synthesized vs. synthesized white noise comparison) is consistent, so we believe it is unlikely that some detail of window construction is responsible for the differences between the studies.

One possible explanation for the difference in asymptotic performance is the smaller pixel pitch of our stimuli: 48.5 pixels per degree, as compared to the 19.7 pixels per degree used in Freeman and Simoncelli, 2011. To investigate this possibility, we down-sampled our target images by half (to a resolution of 1024 by 1300 pixels) using a Gaussian pyramid (via scikit-image’s transform.pyramid_reduce function, Walt et al., 2014; Burt and Adelson, 1987) then synthesized energy model metamers with identical scaling values and optimization hyperparameters to the energy model synthesized vs. synthesized white noise comparison. The psychophysical experiment was run as before, with one subject (sub-00), upsampling the stimuli with nearest neighbor interpolation to present at the same physical size as before, effectively doubling the pixel pitch.

Appendix 4—figure 1

Download asset Open asset

As can be seen in Appendix 4—figure 1, which shows data for a single subject, the critical scaling value did not change, but asymptotic performance increased slightly. For an intuition, see the example stimuli in Appendix 4—figure 2, which show that the patterns used to differentiate the two synthesized stimuli are slightly more obvious with a larger pixel pitch.

Appendix 4—figure 2

Download asset Open asset

Beyond the difference in pixel pitch, there are several other potential factors that may contribute to our lower asymptotic performance:

• Experimental parameters were determined for the original vs. synthesized comparison, which is an easier task than synthesized vs. synthesized comparison. Longer presentation times or some other configuration may increase asymptotic performance in synthesized vs. synthesized comparison (but, as Freeman and Simoncelli, 2011 showed, these sorts of experimental manipulations are unlikely to have much effect on the critical scaling, which is the focus of this study).

• Our stimuli are physically larger, with a diameter of 53.6° compared to 26° in Freeman and Simoncelli, 2011. In debriefing, participants reported performing the task by finding particular informative parts of the stimulus (e.g., high contrast edges) and attending there. Larger stimuli may have made these regions harder to find, and the attentional manipulation in Freeman and Simoncelli, 2011 shows that attending to the most informative region of the stimulus improves asymptotic performance (while leaving critical scaling unchanged). Additionally, Ziemba and Simoncelli, 2021 showed that the probability of correctly differentiating two samples from the same texture family decreases as those samples get larger (conversely, performance increases with stimulus size when participants are differentiating between two different texture families). Something analogous may be happening here, with two synthesized stimuli initialized with white noise acting similarly to two samples from the same texture family.

• As seen both here and in previous studies (Freeman and Simoncelli, 2011; Wallis et al., 2019; Deza et al., 2019), asymptotic performance varies more across target images and subjects than critical scaling. We have a different subset of target images and subjects and so sampling issues may be partly at fault.

All told, there are multiple reasons why asymptotic performance differs between this study and Freeman and Simoncelli, 2011, which we are unable to comprehensively track down.

Appendix 5—figure 1

Download asset Open asset

There are two properties that perceptual metamers must have: they must be perceptually identical and physically distinct. We have shown our synthesized stimuli are perceptually identical and Appendix 5—figure 1 shows that the stimuli are physically distinct. The image at the top of Appendix 5—figure 1 is the original natural image whose representation the model metamers match. The six lower images all have approximately the same mean-squared error to that original image, yet only the luminance model metamer, top left, is confusable with the natural image; the others are easily discriminable from it. The top-right image has had high-frequency noise added uniformly across the image and thus shows the importance of foveation: the luminance model metamer also differs from the original image primarily by the addition of high-frequency noise but, by concentrating the noise in the periphery, it is undetectable.

Appendix 5—figure 2

Download asset Open asset

In addition to global mean-squared error, we can also examine the mean-squared error at each eccentricity, as plotted in Appendix 5—figure 2, which shows the mean-squared error between model metamers and target images as a function of eccentricity for the range of scaling values used in the original vs. synthesized comparison, for the luminance model (A) and the energy model (B). As discussed earlier, when windows get smaller than a pixel, the only value that will have the same model output is the original pixel value. One might have a similar concern for the energy model: what is the smallest pooling window in which a given spatial frequency can be computed and so will windows containing, for example, 16 pixels, also be uniquely constrained when matching six scales? Panel (B) shows this is not the case for our energy model metamers: all scaling values have a non-zero mean-squared error by 1°. However, panel (A) shows that the lowest scaling value for the luminance model metamers has a zero mean-squared error until about 5°, and rises beyond that. While lower scaling values do have a smaller mean-squared error across much of the image, that does not guarantee that they are less informative: mean-squared error is a poor perceptual metric (Wang and Bovik, 2009).

However, when mean-squared error is zero, two stimuli cannot be discriminated between, and so we might worry that the reduced performance for lower scaling values of the luminance model does not reflect the fact that we have found perceptual metamers but that we have removed all information that participants could use to discriminate between the stimuli because of sampling issues with our display. We think this is unlikely for two reasons. First, participants are able to use peripheral information to discriminate between stimuli, including letters (Song et al., 2014), gratings, and Vernier lines (Duncan and Boynton, 2003). Second, our resolution is approximately 48.5 pixels per degree, giving a Nyquist frequency of about 24 cycles per degree. Human grating acuity drops below this frequency by an eccentricity of 2 or 3° (Duncan and Boynton, 2003; Anderson et al., 1991), suggesting that participants would not be able to use this information even if it were present. Additionally, participants reported that the most informative portions of the stimulus were in the mid-periphery, across all conditions, providing additional evidence that they were not relying on the portion of the stimuli where the lowest scaling values were matched to the target stimulus in order to perform the task. Further experiments investigating which portions of the stimulus are the most informative in a more systematic way, such as restricting the stimuli to annuli at different eccentricities and comparing the resulting psychophysical curves, would provide clarity on this matter.

Appendix 6—figures 1 and 2 show the performance for each target image and subject separately, respectively (collapsing over the other dimension). As can also be seen in Figure 10, there’s not much variability in subjects’ critical scaling.

We also see that the two extreme target images (examined in Figure 8) are only extremes for the energy model when comparing against natural and synthesized stimuli: with the exception of the top-right panel, which is replotted from Figure 8, llama and nyc no longer appear as the two ends of the continuum, but lie instead in the middle. In the synthesized vs. synthesized comparisons, the elongated hard edges are not present in either stimulus. For the luminance model original vs. synthesized comparison, elongated contours are no harder to capture than any other natural image feature. In either case, the nyc target is no longer special. Likewise, the llama is not an especially hard comparison for the luminance model, because when the pooling windows get large, the synthesized stimulus contains mostly white noise whereas the original image stimulus contains mostly cloud-like patterns.

We can also see from this figure that the between-image differences are largest for the original vs. synthetic energy model compared to the other three. For the original vs. synthesized luminance model comparison, this is because there is less interaction between the original image features and the model’s invariances: the model is insensitive to high frequencies and thus requires smaller windows to adequately represent that information, but all of our target images are natural images, with $1/f$ power distributions, and thus there is relatively little information present in higher frequencies. For the energy model synthesized vs. synthesized comparison, the two stimuli to distinguish are both synthesized and thus neither contain the edge information that allows subjects to readily distinguish the original from synthesized stimulus in the original vs. synthesized comparison.

Appendix 6—figure 1

Download asset Open asset

Appendix 6—figure 2

Download asset Open asset

The model used to fit our psychophysical curves assumes that each parameter (max $d^{\mathrm{\prime }}$, α, and critical scaling, $s_c$) has a global mean along with subject- and target image-level effects, with no interaction between the two. While this parameterization keeps the total number of parameters very low (29 for each of max $d^{\mathrm{\prime }}$ and critical scaling: 1 for the global mean, 20 for the image-level, and 8 for the subject-level) and allows for a subject to be particularly good at the task (resulting in a higher max $d^{\mathrm{\prime }}$ or a lower critical scaling) or a target image to be particularly difficult, it does not allow for an individual subject to find a target image particularly difficult. To investigate whether this was biasing our results, we fit two additional parameterizations: a completely unpooled model, where max $d^{\mathrm{\prime }}$ and critical scaling were estimated independently for each psychophysical curve (i.e., each model, comparison, subject, and target image) and a partially pooled model with interactions, which takes the model used in the main paper, as described in the ‘Materials and methods’ section (see Equation 3) and adds interaction terms.

That is, for model $m\in \{E,L\}$, comparison $t$, subject $x$, target image $i$, and scaling $s$, the unpooled model fits performance using the parameterization:

with priors:

While the interactions model uses the parameterization:

with the following priors:

Appendix 7—figure 1

Download asset Open asset

A comparison between the three models for all comparisons for which data was collected can be seen in Appendix 7—figure 1, using log-scored leave-one-out cross-validation, estimated using Pareto-smoothed importance sampling (Vehtari et al., 2017; comparable results were obtained using the widely applicable criterion, WAIC, Watanabe, 2013). For the four comparisons for which only a single subject’s data was collected (luminance model: synth vs. synth white noise, energy model: original vs. synth natural image, synth vs. synth natural image, and synth vs. synth white noise [large pixels]), the three models perform comparably. With only a single subject, the models do not differ enough to distinguish their predictions from each other.

Of the remaining three comparisons, the partially pooled model performs best for the two original vs. synth white noise comparisons, while the unpooled model predicts best for the energy model synth vs. synth white noise comparison. Somewhat surprisingly, the interactions model performs slightly worse than the partially pooled model for each comparison, including the one where they are both outperformed by the unpooled model. This may be because the interactions model has far more parameters to fit than the partially pooled model and suggests that the interaction terms are generally unnecessary.

Appendix 7—figure 2

Download asset Open asset

Across all target images and subjects, the unpooled model’s parameter estimates are higher than those of the interactions model, followed by the partially pooled model, though the last two are almost identical (see Appendix 7—figure 1). We will thus discuss the unpooled and partially pooled models. For this comparison, the critical scaling values differ most for the llama, grooming, and terraces images; the others are quite similar. Individual performance curves for both MCMC models are plotted in Appendix 7—figure 2, which may elucidate these differences.

Appendix 7—figure 3

Download asset Open asset

The difference between these two MCMC models arises when there is ambiguous data: when performance data could be fit using either a lower critical scaling and max $d^{\mathrm{\prime }}$ or a higher critical scaling and max $d^{\mathrm{\prime }}$ value, the unpooled model picks the latter, the partially pooled the former. These two parameter regimes make different predictions about subject performance as the synthesis model’s scaling grows beyond the tested values: the unpooled model predicts that performance will continue to increase, while the partially pooled predicts that performance has stabilized. This can be seen most strikingly in the predictions for the llama image: as discussed earlier, this target image is an outlier in its difficulty for all subjects. The two models differ in how they interpret this difficulty: the unpooled model predicts that asymptotic performance is comparable to those of other target images, once we synthesize model metamers with higher scaling values, while the partially pooled model predicts performance will not increase much beyond the observed values. Based on the reasons outlined earlier in this paper, we believe the partially pooled predictions are the most reasonable. This can also be seen in the diverging predictions for sub-06’s performance on the grooming image, where they performed below chance for the highest tested scaling value. The partially pooled model essentially ignores this data point, assuming it’s largely due to lapses, while the unpooled model interprets this performance as requiring a significantly higher critical scaling value. Again, the predictions of the partially pooled model seem more reasonable.

Appendix 7—figure 4

Download asset Open asset

Appendix 7—figure 5

Download asset Open asset

The same pattern can be seen in the other two comparisons. For the luminance model original vs. synth comparison (parameters in Appendix 7—figure 3 and performance in Appendix 7—figure 4), the differences in critical scaling are smaller across MCMC models than in the same comparison for the energy model. However, the same basic pattern appears: unpooled parameter estimates are larger than those of the interactions model, which are larger than those for the partially pooled model.

Appendix 7—figure 6

Download asset Open asset

Appendix 7—figure 7

Download asset Open asset

The largest difference across MCMC models is found in the energy model synth vs. synth comparison (parameters in Appendix 7—figure 5 and performance in Appendix 7—figure 6), which is also the only comparison in which the unpooled model performs better than the partially pooled one. As can be seen by looking at the data in Appendix 7—figure 6, the data quality is poorer than that of the original vs. synth comparisons, which explains the larger discrepancy between the MCMC models’ fitted parameters. However, the unpooled model fit a comparable max $d^{\mathrm{\prime }}$ to the other comparisons (and thus a higher critical scaling), whereas the partially pooled model fits both parameters lower. As discussed in the section ‘Critical scaling is smaller for original vs. synthesized comparisons than synthesized vs. synthesized comparisons’, we believe that asymptotic performance is indeed lower for this comparison and that further increasing the synthesizing model’s scaling will not lead to performances approaching 90% accuracy, as the unpooled model predicts. We thus believe that the partially pooled model’s parameters are more accurate, despite the unpooled model’s better cross-validation performance.

We thus believe that our use of the partially pooled model, and its parameters, is justified in this paper. Furthermore, we should note that none of our conclusions change substantially if using a different MCMC model: in all cases, the luminance model original vs. synth comparison has the smallest critical scaling, followed by the energy model original vs. synth and the energy model synth vs. synth; initializing with a natural image still has an effect on the synth vs. synth critical scaling, but not that of the original vs. synth value. A larger critical scaling value for the energy model synth vs. synth does mean that our value would no longer align with that of Freeman and Simoncelli, 2011, but that is the only change.

Additionally, this model comparison exercise highlights an important point: while our conclusions do not depend on the subjects’ asymptotic performance (as discussed in ’Why does this comparison-type dependency decrease with feature complexity?’, the metamer paradigm makes no predictions for discriminable stimuli), a higher asymptotic performance does make the fitting of the psychophysical curves easier and thus reduces the uncertainty on the inferred parameter values (as we saw that the greatest divergence in parameter values across models comes from curves whose asymptotic performance was poor). As described in Appendix 4, there are many reasons why our asymptotic performance may be lower in this energy model synth vs. synth comparison, some of which may be due to the experimental paradigm. In this paper, we used consistent experimental protocols across all comparisons, but the importance of asymptotic performance to obtaining good quality fits suggests that perhaps future studies should consider varying experimental protocols depending on task difficulty in order to obtain comparable uncertainty in parameter values across comparisons, rather than identical experimental paradigms. Care would need to be taken, however, to ensure that the paradigm does not change so much as to affect the interpretation of the models (such as by, e.g., removing the blank screen between stimulus presentations or allowing free movement of the eyes).

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

The authors would like to thank David Brainard for the use of his photographs, from the UPenn Natural Image Database (Tkačik et al., 2011) and the extended dataset of urban images. They would also like to thank Tony Movshon, David Heeger, David Brainard, Corey Ziemba, and Colin Bredenberg for their feedback on the manuscript, Mike Landy for his assistance with the design of the psychophysical task and feedback on the manuscript, Heiko Schütt for his assistance with the Markov Chain Monte Carlo analysis, and the authors of Wallis et al., 2019 for sharing their code and data. Furthermore, they would like to thank Liz Lovero, Paul Murray, Dylan Simon, and Aaron Watters for their work in creating the metamer browser website.

Human subjects were used in this study. All subjects provided informed consent for taking part in this study, and for its publication. The experiment was conducted in accordance with the Declaration of Helsinki and was approved by the New York University ethics committee on activities involving human subjects.

1. Preprint posted: August 2, 2023
2. Sent for peer review: August 2, 2023
3. Reviewed Preprint version 1: November 1, 2023
4. Reviewed Preprint version 2: October 14, 2025
5. Reviewed Preprint version 3: July 29, 2026
6. Version of Record published: October 15, 2026

You can cite all versions using the DOI https://doi.org/10.7554/eLife.90554. This DOI represents all versions, and will always resolve to the latest one.

© 2023, Broderick et al.

This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited.