Foveated metamers of the early visual system
- Flatiron Institute, Simons Foundation, United States
- Meta, United States
- Department of Psychology, New York University, United States
- Center for Neural Science, New York University, United States
- Courant Inst. for Mathematical Sciences, New York University, United States
Figures
Figure 1
Idealized conceptual schematic of perceptual metamers.
Each panel contains a two-dimensional depiction of the set of all visual images: every image corresponds to a point in this space and every point in this space represents an image. In this and similar figures, each tile represents the set of images which project to the identical point in a perceptual space and hence are metameric, analogous to different lights projecting to the same tri-stimulus coordinates in a color matching space. Any pair of points in different tiles are assumed to project to different locations in a perceptual space; their discriminability depends on the internal signal-to-noise level, analogous to neighboring points in a color matching space. Left: example stimulus (black point), and surrounding metameric stimuli (region enclosed by black polygon). Center: In a hierarchical visual system, in which each stage transforms the signals of the previous stage and discards additional information, every pair of stimuli that are metamers for an early visual area are also metamers for the later visual area . Thus, metamers of earlier stages are nested within metamers of later stages. The converse does not necessarily hold: there may be stimuli that can distinguish but that cannot. Right: Two stimulus families used for initialization of our metamer synthesis algorithm: white noise (distribution represented as a grayscale intensity map in the lower right corner) and natural images (distribution represented by the curved gray manifold). Typical white noise samples fall within a single perceptual metamer class (humans are unable to distinguish them). Natural images, on the other hand, span many tiles in the perceptual space, and are generally distinguishable from each other.
Figure 2
Two pooling models.
Both models compute local image statistics within a Gaussian-weighted window that is separable in a log-polar coordinate system, such that radial extent is approximately twice the angular extent (red contours indicate half-maximum levels). Windows are uniformly spaced in a log-polar grid, with centers separated by one standard deviation. A single scaling factor governs the size of all pooling windows. The luminance model (left) computes average luminance. The spectral energy model (right) computes average spectral energy at 4 orientations and 6 scales, as well as luminance, for a total of 25 statistics per window. Spectral energy is computed using the complex steerable pyramid constructed in the Fourier domain (Simoncelli and Freeman, 1995), squaring and summing across the real and imaginary components. Full-resolution version of this figure can be found on the OSF.
Figure 3
Metamers in human perception and pooling models.
Panels depict a two-dimensional stimulus space, with polygonal regions indicating groups of perceptually indistinguishable stimuli (human metamers, see Figure 1). Pooling models determine the statistics that are pooled, with pooling extent controlled by scaling parameter, . Ellipses indicate model metamers: sets of stimuli that are physically different but whose model outputs are identical to those of the target (original) image . We generate model metamer samples using an optimization procedure: starting from initial image we adjust the pixel values until their pooled statistics match those of the target. The shapes and sizes of metameric regions (ellipses) depend on the model (), the scaling parameter (), the statistics of the target image , as well as the initial images () and the synthesis algorithm. For a given set of statistics, increasing the scaling value by factor increases the size of the metamer set, and any stimulus that is a metamer for will also be a metamer for (i.e., larger desaturated ellipses contain the smaller saturated ellipses). Critical scaling is the largest for which all model metamers are human metamers (smaller ellipses). Left: Model with parameters that produce approximate alignment with human metamers. Right: Model with different pooled statistics that yield metamers (blue ellipses) that are poorly aligned with human metamers: at critical scaling, there are human metamers that are not model metamers. This mismatch cannot be resolved by adjusting the scaling parameter: increasing such that all human metamers are also model metamers (larger ellipse) will also yield model metamers that are not human metamers. See also Feather et al., 2023, Figure 1.
Figure 4
Example synthesized model metamers.
Top: Target image. Middle: Luminance model metamers, computed for two different scaling values (values as indicated, red ellipses to right of fixation indicate pooling window contours at half-max at that eccentricity). The left image is computed with a small scaling value, and is a perceptual metamer for most subjects: when fixating at the cross in the center of the stimulus, the two stimuli appear perceptually identical to the target image. Note, however, that when fixating in the periphery (e.g., the blue box), one can clearly see that the stimulus differs from the target (see enlarged versions of the foveal and peripheral neighborhoods to the right). The right image is computed with a larger scaling value and is no longer a perceptual metamer (for any choice of observer fixation). Bottom: Energy model metamers. Again, the left image is computed with a small scaling value and is a perceptual metamer for most observers when fixated on the center cross. Peripheral content (e.g., blue box) contains more complex distortions, readily visible when viewed directly. The right image, computed with a large scaling value, differs in appearance from the target regardless of observer fixation. Full-resolution version of this figure can be found on the OSF.
Figure 5
Target images used in the experiments.
Images contain a variety of content, including textures, objects, and scenes. All are RAW camera images, with values proportional to luminance and quantized to 16 bits. Images were converted to grayscale, cropped to 2048×2600 pixels, displayed at 53.6×42.2°, with intensity values rescaled to lie within the range of [0.05, 0.95] of the display intensities. All subjects saw target images 1–10, half saw 11–15, and half saw 16–20. A full-resolution version of this figure can be found on the OSF.
Figure 6
Performance curves and their corresponding parameter values for different models and stimulus comparisons.
The luminance model has a substantially smaller critical scaling than the energy model, and original vs. synthesized comparisons yield smaller critical scaling values than synthesized vs. synthesized comparisons. (A) Psychometric functions, expressing probability correct as a function of scaling parameter, for both energy and luminance models (aqua and beige, respectively), and original vs. synthesized (solid line) and synthesized vs. synthesized (dashed line) comparisons. Data points represent average values across subjects and target images, 4320 trials per data point except for the luminance model synthesized vs. synthesized comparison, which has only 180 trials per data point (one subject, five target images). Lines represent the posterior predictive means of fitted curves across subjects and target images, with the shaded region indicating the 95% highest density interval (HDI, Kruschke, 2015). (B) Estimated parameter values, separated by target image (left) or subject (right). Top row shows the critical scaling value and the bottom the value of the maximum parameter. Points represent the posterior means, shaded regions the 95% HDI, and horizontal dashed lines and shaded regions the global means and 95% HDI. Note that the luminance model, synthesized vs. synthesized comparison is not shown, because the data are poorly fit (A, beige dashed line).
Figure 7
Comparison of two synthesized metamers is more difficult than comparison of a synthesized metamer with the original image.
For the highest tested scaling value (1.5) the original vs. synthesized comparison is trivial while the synthesized vs. synthesized comparison is difficult (energy model) or impossible (luminance model). Top: target image. Middle: Two luminance model metamers, generated from different initial uniform noise images. Bottom: Two energy model metamers, generated from different initial uniform noise images. All four of the model metamers can be easily distinguished from the natural image at top (original vs. synthesized), but are difficult to distinguish from each other, despite the fact that their pooling windows have grown very large (synthesized vs. synthesized). Full-resolution version of this figure can be found on the OSF.
Figure 8
The interaction between image content and model sensitivities greatly affects asymptotic performance (especially for the synthesized vs.synthesized comparison using the energy model) while critical scaling does not vary as much.
(A) Performance for each target image, averaged across subjects, comparing synthesized stimuli to natural image stimuli. Most target images show similar performance, with one obvious outlier whose performance never rises above 60%. Data points represent the average across subjects, 288 trials per data point for half the images, 144 per data point for the other half. Lines represent the posterior predictive means across subjects, with the shaded region giving the 95% HDI. (B) Example energy model metamers for two extreme target images. The top row (nyc) is the target image with the best performance (purple line in panel A), while the bottom row (llama) has the worst performance (red line in panel A). In each row, the leftmost image is the target image, and the next two show model metamers with the lowest and highest tested scaling values for this comparison. Full-resolution version of this figure can be found on the OSF.
Figure 9
Initializing model metamers with natural images does not affect performance in the original vs. synthesized comparison, but reduces critical scaling and increases max for the synthesized vs. synthesized comparison.
Note all data in this figure is for a single subject and 15 of the 20 target images. (A) Probability correct for one subject (sub-00), as a function of scaling. Each point represents the average of 540 trials (over all fifteen target images), except for the synthesized vs. synthesized luminance model white noise comparison (averaged over 5 target images). Vertical black line indicates scaling value where performance ran from chance to 100%, based on initialization and comparison, as discussed in panel (B). (B) Three comparisons corresponding to the three psychophysical curves intersected by the vertical black line in panel (A). The small image next to each model metamer shows the image used to initialize synthesis. See text for details. Full-resolution version of this figure can be found on the OSF.
Figure 10
Parameter values for the comparisons shown in Figure 9A (top: critical scaling value; bottom: max ).
Data shown is from the single subject who completed all comparisons. Points represent the posterior means, shaded regions the 95% HDI, and horizontal dashed lines and shaded regions average across all shown stimuli for this subject. Note that the luminance model, synthesized vs. synthesized: white noise comparison is not shown in this panel because the data was poorly fit by this curve.
Figure 11
Critical scaling values for the two pooling models presented in this paper (Luminance and Energy) along with a Texture model (data from Freeman and Simoncelli, 2011 and Wallis et al., 2019, averaging across the two image classes).
Filled points indicate original vs. synthesized comparisons, while unfilled points indicate synthesized vs. synthesized comparisons (for the luminance model, this is effectively infinite, since participants were unable to perform the task for any scaling value). For all three models, critical values are smaller in the original vs. synthesized comparison than the synthesized vs. synthesized one, but this effect decreases with increasing complexity of image statistics. Our critical scaling values for synth vs. synth comparisons of the energy model are consistent with those reported by Freeman and Simoncelli, 2011.
Figure 12
Illustration of how critical scaling value can depend on comparison type.
Left: For the original vs. synthetic comparison, critical scaling corresponds to the largest ellipse such that the synthetic image is indistinguishable from (i.e., lies within the same perceptual region as) the target (original) image . Right: Example configuration in which synthetic vs. synthetic comparisons lead to a larger critical scaling (with correspondingly larger ellipse). The two synthesized images are indistinguishable from each other but not from the target.
Figure 13
Illustration of how discrepancies between the original vs. synth and synth vs. synth comparisons can be model-dependent.
Upper panels depict a hypothetical luminance model, for which pairs of synthesized stimuli are indistinguishable for all scaling values (i.e., they lie within the same perceptual metamer class for both the small ellipse on the left, and for the larger ellipse on the right). In this case, critical scaling cannot be estimated (indicated by dashes in ellipse). Lower panels depict a hypothetical texture model, for which synth vs. synth comparisons yield a critical scaling value (right) only slightly larger than that obtained for original vs. synth (left). Energy model metamers (not shown) lie between these two extremes (see Figure 11).
Figure 14
Illustration of how discrepancies between original vs. synth and synth vs. synth comparisons can depend on the image used to initialize synthesis.
Initializing with white noise (as is commonly done) can lead to a large difference between the two comparisons (see also Figures 12 and 13). Initializing with a natural image can reduce the magnitude of the difference.
Figure 15
Schematic of psychophysics task.
Top shows the structure for a single trial: a single stimulus is presented for 200 msec, bisected in the center by a gray bar, followed by a blank screen for 500 msec. The stimulus is re-displayed, with a random half of the stimulus changed to the comparison stimulus, for 200 msec. The participants then have as long as needed to say which half of the stimulus changed, followed by a 500 msec intertrial interval. Bottom table shows possible comparisons. In original vs. synthesized, one stimulus was the target image stimulus whose model representation the synthesized stimuli match (see Figure 5), and the other was one such synthesized stimulus. In synthesized vs. synthesized, both were synthesized stimuli targeting the same target image stimulus, with the same model scaling, but different initialization. In experiments, dividing bar, blanks, and backgrounds were all midgray. For more details, see text.
Figure 16
Relationship between the max parameter, α, and asymptotic performance.
As max increases beyond approximately 5 (where asymptotic performance is at ceiling), the slope of the psychophysical curve continues to increase (e.g., compare the slope of the luminance and energy model original vs. synth white noise comparisons in Figure 9A).
Appendix 1—figure 1
With luminance model metamers synthesized with scaling value 0.23, performance depends strongly on the comparison.
We only have data for the comparison in the bottom row (synthesized vs. synthesized white noise), where performance is at chance. From inspection, we believe performance would be at ceiling for either the original vs. synthesized or synthesized vs. synthesized comparisons when initializing with natural images (top right: initialized with lettuce, middle left: nyc, middle right: terrace). Insets show the images used to initialize metamer synthesis. Full-resolution version of this figure can be found on the OSF.
Appendix 2—figure 1
Examples of synthesized luminance model metamers, generated from different initial image types, each with scaling value 0.045.
Discriminability from the target image depends on the initialization. For this scaling value, performance was at ceiling when comparing the white noise-initialized model metamer (bottom left) to the target image (top left), as can be seen in Figure 6A in the main text. However, the pink noise-initialized metamer (bottom right) lacks the high frequencies used to perform this discrimination; from inspection, we believe performance would be at or near chance. From inspection, we believe performance for the natural image-initialized metamer (top right) would be intermediate between the two. Insets show the images used to initialize metamer synthesis. Full-resolution version of this figure can be found on the OSF.
Appendix 3—figure 1
Scaling has the same interpretation for the Gaussian windows of this study and the raised-cosine windows used in previous studies.
The left column shows the windows from Freeman and Simoncelli, 2011, the center column shows the windows from this paper, and the right column shows their difference. The windows have been normalized so as they both have a maximum value of 1 (when used in our models, the Gaussian windows have a lower maximum value so that the sum across all windows is 1). While the centers and extent of the two sets of windows are different, their full-width at half-maximum is matched.
Appendix 4—figure 1
Asymptotic performance on the synthesized vs. synthesized white noise comparison is slightly higher when stimuli have a larger pixel pitch.
(A) Probability correct for one subject, sub-00, as a function of scaling for energy and luminance models. Data points represent the average across target images, 540 trials per data point (one subject, 15 target images) except for luminance model synthesized vs. synthesized: white noise comparison, which have 180 per data point (one subject, five target images). Lines represent the posterior predictive means across target images, with the shaded region giving the 95% HDI. (B) Parameter values for these comparisons. Top row shows the critical scaling value and the bottom the value of the max parameter. Left column presents the values for each target image separately for this one subject, while the right presents the values for this subject, averaged across target images. In this case, we only present the data for sub-00, as they are the only subject to perform the task with larger pixel pitch. Points represent the posterior means, shaded regions the 95% HDI, and horizontal dashed lines and shaded regions average across all shown target images for this subject. Note that the luminance model, synthesized vs. synthesized: white noise comparison is not shown in this panel, because the data was poorly fit by this curve—as can be seen in panel (A), the psychophysical curve is essentially flat at chance and thus the fit had low max and low critical scaling, with high uncertainty.
Appendix 4—figure 2
Energy model metamers synthesized with larger pixel pitch are slightly easier to distinguish from each other at high scaling values.
Top row shows the target image stimulus whose representation all presented model metamers were synthesized to match. Middle row shows two energy model metamers initialized with different patches of white noise, which were shown in the synthesized vs. synthesized white noise comparison. Bottom row shows two model metamers initialized with the same settings but half the pixel resolution, which were shown in the large pixel version of the synthesized vs. synthesized white noise comparison. As these stimuli have half the resolution in each direction as the middle row but were presented at the same physical display size, their pixel pitch was doubled, making it comparable to that of Freeman and Simoncelli, 2011. With the larger pixel pitch, the snake-like patterns that can be used to distinguish the two synthesized stimuli have a lower spatial frequency (in cycles per degree), which, given their presence in the periphery where spatial frequency sensitivities are lower, may account for the participant’s increased ability to distinguish such stimuli. Full-resolution version of this figure can be found on the OSF.
Appendix 5—figure 1
Luminance model metamers are substantially physically different from their target images.
The six bottom stimuli all have approximately the same mean-squared error to the top image, the target natural image. Notice that the luminance model metamer synthesized with scaling value 0.01 (top left) is by far the least discriminable with the natural image when fixating on the center, the others are obviously distinct. This pair of original and synthesized images was chosen to set the MSE because it has the lowest MSE for all luminance metamers when compared with their target image. Full-resolution version of this figure can be found on the OSF.
Appendix 5—figure 2
Mean-squared error (MSE) between model metamers and target images as a function of eccentricity, averaged radially, for the luminance model (A) and energy model (B).
MSE is computed between images represented as 8-bit unsigned integers. Each scaling value is shown as a separate color, and each target image is plotted on a separate axis. For the luminance model, the smallest scaling value has zero MSE out to about 5°, and all scaling values rise with eccentricity and then saturate. For the energy model, all scaling values have non-zero MSE by 1° and are indistinguishable beyond that for most images, though for some, such as gnarled and ivy, lower scaling values have higher MSE across eccentricities.
Appendix 6—figure 1
Image content only matters for the energy model, original vs. synthesized comparison.
Performance for each target image, averaged across subjects, is plotted separately. This is because the image content affects performance based on how it interacts with the models’ sensitivities and insensitivities: llama contains little phase structure and is thus easy for the energy model to capture even at larger scaling values, while nyc contains many hard edges, and is thus difficult for the energy model to capture even with the smallest tested windows. However, for the luminance model original vs. synthesized comparison, the relevant interaction would be with the Fourier spectra: the model is insensitive to high frequencies in its periphery. As all images have spectra everywhere in the image, none are outliers in terms of difficulty. For the synthesized vs. synthesized comparisons, on the other hand, by definition, neither displayed stimulus contains those features that the model is insensitive to, which is what enables correct performance in the original vs. synthesized case. Thus, stimuli are more interchangeable with each other. Colors and symbols are the same as in Figure 6, except for the two lines showing the nyc (purple) and llama (red) target images, which are colored as in Figure 8. Top-right subplot is reprinted from Figure 8A. Note that the bottom-left subplot only contains five target images (and in particular does not include the purple line representing nyc), as only a single session of that comparison was run.
Appendix 6—figure 2
Performance does not vary much across subjects, and varies more in terms of max than critical scaling.
Each line represents the performance for a single subject, averaged across target images. Colors and symbols are the same as in Figure 6. Note that the bottom-left subplot only contains a single subject, since only one subject completed this comparison.
Appendix 7—figure 1
Comparison of models used to fit psychophysical curves, using log-scored leave-one-out cross-validation, estimated using Pareto-smoothed importance sampling (Vehtari et al., 2017).
A higher score corresponds to better predictive accuracy. Each model was fit separately to each model and comparison. The models differ in how parameters are modeled across subjects and target images: the ‘unpooled’ model assumes no relationship across subjects or target images, the ‘partially pooled’ model has separate image- and subject-level effects for each parameter, and the ‘partially pooled-interactions’ model adds interaction terms to that model.
Appendix 7—figure 2
Comparison of parameters for the energy model, original vs. synth white noise comparison, averaged across subjects and target images, for each of the three MCMC models.
The partially pooled values are identical to those shown in Figure 6. The values for the unpooled model are always greater than or equal to those for the interactions model, which are almost identical to those for the partially pooled model (though they are, as a rule, slightly larger).
Appendix 7—figure 3
Comparison between psychophysical curves fit by the partially pooled (A) and unpooled (B) MCMC models for the energy model original vs. synth comparison.
The two models diverge the most when performance data appears ambiguous and, in general, the unpooled model fits higher values for both the max and critical scaling.
Appendix 7—figure 4
Comparison of parameters for the luminance model, original vs. synth white noise comparison, averaged across subjects and target images, for each of the three MCMC models.
The partially pooled values are identical to those shown in Figure 6. The values for the unpooled model are always greater than or equal to those for the interactions model, which are almost identical to those for the partially pooled model (though they are, as a rule, slightly larger).
Appendix 7—figure 5
Comparison between psychophysical curves fit by the partially pooled (A) and unpooled (B) MCMC models for the luminance model original vs. synth comparison.
The two models diverge the most when performance data appears ambiguous and, in general, the unpooled model fits higher values for both the max and critical scaling.
Appendix 7—figure 6
Comparison of parameters for the energy model, synth vs. synth white noise comparison, averaged across subjects and target images, for each of the three MCMC models.
The partially pooled values are identical to those shown in Figure 6. The values for the unpooled model are always greater than or equal to those for the interactions model, which are almost identical to those for the partially pooled model (though they are, as a rule, slightly larger).
Appendix 7—figure 7
Comparison between psychophysical curves fit by the partially pooled (A) and unpooled (B) MCMC models for the energy model synth vs. synth comparison.
The two models diverge the most when performance data appears ambiguous and, in general, the unpooled model fits higher values for both the max and critical scaling.
Tables
Table 1
Critical scaling (posterior mean over all subjects and target images) and number of statistics (as a percentage of number of input image pixels—a type of ‘compression rate’) for each model and comparison.
Note that the critical scaling for original vs. synthesized comparisons does not result in the same number of statistics across models and, in particular, at their critical scaling values, all models have dimensionality less than that of the input image.
| Model | Comparison | Critical scaling | Number of statistics (percentage of image pixels) |
|---|---|---|---|
| Luminance | Original vs. synth: white noise | 0.017 | 19.6% |
| Synth vs. synth: white noise | N/A | N/A | |
| Energy | Original vs. synth: white noise | 0.065 | 34.7% |
| Original vs. synth: natural image | 0.068 | 31.5% | |
| Synth vs. synth: natural image | 0.114 | 11.6% | |
| Synth vs. synth: white noise | 0.252 | 2.6% |
Additional files
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Foveated metamers of the early visual system
eLife 12:RP90554.
https://doi.org/10.7554/eLife.90554.4
