An image reconstruction framework forcharacterizing initial visual encoding

Abstract
Data availability
Article and author information
Metrics

Abstract

We developed an image-computable observer model of the initial visual encoding that operates on natural image input, based on the framework of Bayesian image reconstruction from the excitations of the retinal cone mosaic. Our model extends previous work on ideal observer analysis and evaluation of performance beyond psychophysical discrimination, takes into account the statistical regularities of the visual environment, and provides a unifying framework for answering a wide range of questions regarding the visual front end. Using the error in the reconstructions as a metric, we analyzed variations of the number of different photoreceptor types on human retina as an optimal design problem. In addition, the reconstructions allow both visualization and quantification of information loss due to physiological optics and cone mosaic sampling, and how these vary with eccentricity. Furthermore, in simulations of color deficiencies and interferometric experiments, we found that the reconstructed images provide a reasonable proxy for modeling subjects' percepts. Lastly, we used the reconstruction-based observer for the analysis of psychophysical threshold, and found notable interactions between spatial frequency and chromatic direction in the resulting spatial contrast sensitivity function. Our method is widely applicable to experiments and applications in which the initial visual encoding plays an important role.

Data availability

The MATLAB code used for this paper is available at: https://github.com/isetbio/ISETImagePipelineIn addition, the curated RGB and hyperspectral image datasets, parameters used in the simulation including display and cone mosaic setup, as well as the intermediate results such as the learned sparse priors, likelihood functions (i.e., render matrices), are available through: https://tinyurl.com/26r92c8y

The following previously published data sets were used

Article and author information

Author details

Ling-Qi Zhang

Department of Psychology, University of Pennsylvania, Philadelphia, United States

For correspondence
lingqiz@sas.upenn.edu

Competing interests
Ling-Qi Zhang, Funding provided by Facebook Reality Labs.

"This ORCID iD identifies the author of this article:" 0000-0001-8468-7927
Nicolas P Cottaris

Department of Psychology, University of Pennsylvania, Philadelphia, United States

Competing interests
Nicolas P Cottaris, Funding provided by Facebook Reality Labs.
David Brainard

Department of Psychology, University of Pennsylvania, Philadelphia, United States

Competing interests
David Brainard, Funding provided by Facebook Reality Labs.

"This ORCID iD identifies the author of this article:" 0000-0001-9827-543X

Funding

Facebook Reality Labs

Ling-Qi Zhang
Nicolas P Cottaris
David Brainard

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Copyright

This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.