The ability to assess how tilted a surface is, or its ‘surface orientation’, is critical for interacting productively with the environment. For example, it helps organisms to determine whether a particular surface is better suited for walking or climbing. Humans and other animals estimate 3-dimensional (3D) surface orientations from 2-dimensional (2D) images on their retinas. But exactly how they calculate the tilt of a surface from the retinal images is not well understood.

Scientists have studied how humans estimate surface orientation by showing them smooth (often planar) surfaces with artificial markings. These studies suggested that humans very accurately estimate the direction in which a surface is tilted. But whether humans are as good at estimating surface tilt in the real world, where scenes are more complex than those tested in experiments, is unknown.

Now, Kim and Burge show that human tilt estimation in natural scenes is often inaccurate and imprecise. To better understand humans’ successes and failures in estimating tilt, Kim and Burge developed an optimal computational model, grounded in natural scene statistics, that estimates tilt from natural images. Kim and Burge found that the model accurately predicted how humans estimate tilt in natural scenes. This suggests that the imprecise human estimates are not the result of a poorly designed visual system. Rather, humans, like the computational model, make the best possible use of the information images provide to perform an estimation task that is very difficult in natural scenes.

The study takes an important step towards generalizing our understanding of human perception from the lab to the real world.

Understanding how vision works in natural conditions is a primary goal of vision research. One measure of success is the degree to which performance in a fundamental visual task can be predicted directly from image data. Estimating the 3D structure of the environment from 2D retinal images is just such a task. However, relatively little is known about how the human visual system estimates 3D surface orientation from images of natural scenes.

3D surface orientation is typically parameterized by slant and tilt. Slant is the amount by which a surface is rotated away from an observer; tilt is the direction in which the surface is rotated (Figure 1A). Compared to slant, tilt has received little attention, even though both are critically important for successful interaction with the 3D environment. For example, even if slant has been accurately estimated, humans must estimate tilt to determine where they can walk. Surface with tilts of 90°, like the ground plane, can sometimes be walked on. Surfaces with tilts of 0° or 180°, like the sides of tree trunks, can never be walked on.

Figure 1

Download asset Open asset

Numerous psychophysical, computational, and neurophysiological studies have probed the human ability to estimate surface slant, surface tilt, and 3D shape. Systematic performance has been observed, and models have been developed that nicely describe performance. Most previous studies have used stimuli having planar (Stevens, 1983;Knill, 1998a, 1998b; Hillis et al., 2004; Burge et al., 2010a; Rosenholtz and Malik, 1997; Rosenberg et al., 2013; Murphy et al., 2013; Velisavljević and Elder, 2006; Saunders and Knill, 2001; Welchman et al., 2005; Sanada et al., 2012; Tsutsui et al., 2001) or smoothly curved (Todd et al., 1996; Fleming et al., 2011; Todd, 2004;Marlow et al., 2015; Li and Zaidi, 2000, 2004; Norman et al., 2006) surface shapes and regular (Knill, 1998a, 1998b; Hillis et al., 2004; Watt et al., 2005; Rosenholtz and Malik, 1997; Rosenberg et al., 2013; Murphy et al., 2013; Velisavljević and Elder, 2006; Li and Zaidi, 2000, 2004; Welchman et al., 2005) or random-patterned (Burge et al., 2010a; Fleming et al., 2011) surface markings. These stimuli are not representative of the variety of surface shapes and markings encountered in natural viewing. Surfaces in natural scenes often have complex surface geometries and are marked by complicated surface textures. Thus, performance with simple artificial scenes may not be representative of performance in natural scenes. Also, models developed with artificial scenes often generalize poorly (or cannot even be applied) to natural scenes. These issues concern not just studies of 3D surface orientation perception but vision and visual neuroscience at large.

Few studies have examined the human ability to estimate 3D surface orientation using natural photographic images, the stimuli that our visual systems evolved to process. None, to our knowledge, have done so with high-resolution groundtruth surface orientation information. There are good reasons for this gap in the literature. Natural images are complex and difficult to characterize mathematically, and groundtruth data about natural scenes are notoriously difficult to collect. Research with natural stimuli has often been criticized (justifiably) on the grounds that natural stimuli are too complicated or too poorly controlled to allow strong conclusions to be drawn from the results. The challenge, then, is to develop experimental methods and computational models that can be used with natural stimuli without sacrificing rigor and interpretability.

Here, we report an extensive examination of human 3D tilt estimation from local image information with natural stimuli. We sampled thousands of natural image patches from a recently collected stereo-image database of natural scenes with precisely co-registered distance data (Figure 1B) (Burge et al., 2016). Groundtruth surface orientation was computed directly from the distance data (see Materials and methods). Human observers binocularly viewed the natural patches and estimated the tilt at the center of each patch. The same human observers also viewed artificially-textured planar stimuli matched to the groundtruth tilt, slant, distance, and luminance contrast of the natural stimuli. First, we compared human performance with natural and matched artificial stimuli. Then, we compared human performance to the predictions of an image-computable normative model, a Bayes’ optimal observer, that makes the best possible use of the available image information for the task. This experimental design enables direct, meaningful comparison of human performance across stimulus types, allowing the isolation of important stimulus differences and the interpretation of human response patterns with respect to principled predictions provided by the model.

A rich set of results emerges. First, tilt estimation in natural scenes is hard; compared to performance with artificial stimuli, performance with natural stimuli is poor. Second, with natural stimuli, human tilt estimates cluster at the cardinal tilts (0°, 90°, 180° and 270°), echoing the prior distribution of tilts in natural scenes (Figure 1C) (Burge et al., 2016; Yang and Purves, 2003a;Yang and Purves, 2003b; Adams et al., 2016). Third, human estimates tend to be more biased and variable when the groundtruth tilts are oblique (e.g., 45°). Fourth, at each groundtruth tilt, the distributions of human and model errors tend to be very similar, even though the error distributions themselves are highly irregular. Fifth, human and model observer trial-by-trial errors are correlated, suggesting that similar (or strongly correlated) stimulus properties drive both human and ideal performance. Together, these results represent an important step towards the goal of being able to predict human percepts of 3D structure directly from photographic images in a fundamental natural task.

Human observers binocularly viewed thousands of randomly sampled patches of natural scenes; they viewed an equal number of stimuli at each of 24 tilt bins between 0° and 360°. The stimuli were presented on a large (2.0 × 1.2 m) stereo front-projection system positioned 3 m from the observer. This relatively long viewing distance minimizes focus cues to flatness. Except for focus cues, the display system recreates the retinal images that would have been formed by the original scene. Each scene was viewed binocularly through a small virtual aperture (1° or 3° of visual angle) positioned 5 arcmin of disparity in front of the sampled point in the scene (Figure 2A); the viewing situation is akin to looking at the world through a straw (McDermott, 2004). Patches were displayed at the random image locations from which they were sampled. Observers reported, using a mouse-controlled probe, the estimated surface tilt at the center of each patch (Figure 2B). We pooled data across human observers and aperture sizes and converted the tilt estimates to unsigned tilt for analysis (signed tilt modulo 180°) because the estimation of unsigned tilt was similar for all observers and aperture sizes (Figure 2—figure supplement 1, Figure 2—figure supplement 2). The same observers also estimated surface tilt with an extensive set of artificial planar stimuli that were matched to the tilts, slants, distances, and luminance contrasts of the natural stimuli presented in the experiment. (Each planar artificial stimulus had one of three texture types: 1/f noise, 3.5 cpd plaid, and 5.25 cpd plaid; Figure 2—figure supplement 3.) Thus, any observed performance differences between natural and artificial stimuli cannot be attributed to these dimensions.

Figure 2 with 5 supplements see all

Download asset Open asset

Natural and artificial stimuli elicited strikingly different patterns of performance (Figure 2C). Although many stimuli of both types elicit tilt estimates ${\displaystyle \hat{\tau }}$ that approximately match the groundtruth tilt (data points on the unity line), a substantial number of natural stimuli elicit estimates that cluster at the cardinal tilts (data points at ${\displaystyle \hat{\tau }=0^{\circ },{90}^{\circ },{180}^{\circ },{270}^{\circ }}$). No such clustering occurs with artificial stimuli. The histogram of the human tilt estimates explicitly shows the clustering, or lack thereof (Figure 2D). With natural stimuli, the distribution of unsigned estimates ${\displaystyle p(\hat{\tau })}$ peaks at 0° and 90° and has a similar shape to the prior distribution of groundtruth tilts in the natural scene database (Figure 1C; also see Figure 2—figure supplement 4). If the database is representative of natural scenes, then one might expect the human visual system to use the natural statistics of tilt as a tilt prior in the perceptual processes that convert stimulus measurements into estimates. Standard Bayesian estimation theory predicts that the prior will influence estimates more when measurements are unreliable and will influence estimates less when measurements are reliable (Knill and Richards, 1996).

We summarized 3D tilt estimation performance by computing the mean and variance of the tilt estimates ${\displaystyle \hat{\tau }}$ as a function of groundtruth tilt (Figure 2E,F). (The mean and variance were computed using circular statistics because tilt is an angular variable; see Materials and methods.) These summary statistics change systematically with groundtruth tilt, exhibiting patterns reminiscent of the 2D oblique effect (Appelle, 1972; Furmanski and Engel, 2000; Girshick et al., 2011). With natural stimuli, estimates are maximally biased at oblique tilts and unbiased at cardinal tilts; estimate variance is highest at oblique tilts (~60° and ~120°) and lowest at cardinal tilts. With artificial stimuli, estimates are essentially unbiased and are less variable across tilt. The unbiased responses to artificial stimuli imply that the biased responses to natural stimuli accurately reflect biased perceptual estimates, under the assumption that the function that maps perceptual estimates to probe responses is stable across stimulus types (see Materials and methods). (See Figure 2—figure supplement 3 for performance with each individual artificial texture type.) The summary statistics reveal clear differences between the stimulus types. However, there is more to the data than the summary statistics can reveal. Thus, we analyzed the raw data more closely.

The probabilistic relationship between groundtruth tilt $\tau$ and human tilt estimates ${\displaystyle \hat{\tau }}$ is shown in Figures 3 and 4. Each subplot in Figure 3A shows the distribution of estimation errors ${\displaystyle p\left(\hat{\tau }-\tau |\tau \right)}$ for a different groundtruth tilt. With artificial stimuli, estimation errors ${\displaystyle \mathbf{e}=\hat{\tau }-\tau }$ are unimodally distributed and peaked at zero (black symbols). With natural stimuli, estimation errors are more irregularly distributed, and the peak locations change systematically with groundtruth tilt (white points). With cardinal groundtruth tilts (e.g., ${\displaystyle \tau =0^{\circ }}$ or ${\displaystyle \tau ={90}^{\circ }}$), the error distributions peak at zero and large errors are rare. With oblique groundtruth tilts (e.g., ${\displaystyle \tau ={60}^{\circ }}$ or ${\displaystyle \tau ={120}^{\circ }}$), the error distributions tend to be bi-modal with two prominent peaks at non-zero errors. For example, when groundtruth tilt ${\displaystyle \tau ={60}^{\circ }}$, the most common errors were −60° and 30°. These errors occurred because observers incorrectly estimated the tilt to be 0° or 90°, respectively, when the correct answer was 60º. Thus, at this groundtruth tilt, the human observers frequently (and incorrectly) estimated cardinal tilts instead of the correct oblique tilt.

Figure 3

Download asset Open asset

Figure 4 with 1 supplement see all

Download asset Open asset

Tilt estimates from natural stimuli are less accurate at oblique than at cardinal groundtruth tilts. Does this fact imply that oblique tilt estimates (e.g., ${\displaystyle \hat{\tau }={60}^{\circ }}$) provide less accurate information about groundtruth tilt than cardinal tilt estimates (e.g., ${\displaystyle \hat{\tau }={90}^{\circ }}$)? No. Each panel in Figure 4A shows the distribution of groundtruth tilts ${\displaystyle p\left(\tau |\hat{\tau }\right)}$ for each estimated tilt. The most probable groundtruth tilt equals the estimated tilt, and the variance of each distribution is approximately constant, regardless of whether the estimated tilt is cardinal or oblique. Furthermore, the estimates from natural and artificial stimuli provide nearly equivalent information about groundtruth (see also Figure 4—figure supplement 1). Thus, even though tilt estimation performance is far poorer at oblique than at cardinal tilts and is far poorer with natural than with artificial stimuli, all tilt estimates regardless of the value are similarly good predictors of groundtruth tilt.

How can it be that low-accuracy estimates from natural stimuli predict groundtruth nearly as well as high-accuracy estimates from artificial stimuli? Some regions of natural scenes yield high-reliability measurements that make tilt estimation easy; other regions of natural scenes yield low-reliability measurements that make tilt estimation hard. When measurements are reliable, the prior influences estimates less; when measurements are unreliable, the prior influences estimates more. Thus, cardinal tilt estimates can result either from reliable measurements of cardinal tilts or from unreliable measurements of oblique tilts. On the other hand, oblique tilt estimates can only result from reliable measurements of oblique tilts, because the measurements must be reliable enough to overcome the influence of the prior. All these factors combine to make each tilt estimate, regardless of its value, an equally reliable predictor of groundtruth tilt. The uniformly reliable information provided by the estimates about groundtruth (see Figure 4A) may simplify the computational processes that optimally pool local estimates into global estimates (see Discussion). The generality of this phenomenon across natural tasks remains to be determined. However, we speculate that it may have widespread importance for understanding perception in natural scenes, as well as in other circumstances where measurement reliability varies drastically across spatial location.

Normative model

We asked whether the complicated pattern of human performance with natural stimuli is consistent with optimal information processing. To answer this question, we compared human performance to the performance of a normative model, a Bayes optimal observer that optimizes 3D tilt estimation in natural scenes given a squared error cost function (Burge et al., 2016). The model takes three local image cues ${\displaystyle \mathbf{C}}$ as input — luminance, texture, and disparity gradients — and returns the minimum mean squared error (MMSE) tilt estimate ${\displaystyle {\hat{\tau }}_{MMSE}}$ as output. (The MMSE estimate is the mean of the posterior probability distribution over groundtruth tilt given the measured image cues.)

To determine the optimal estimate for each possible triplet of cue values, we use the natural scene database. At each pixel in the database, the image cues are computed directly from the photographic images within a local area, and the groundtruth tilt is computed directly from the distance data (see Materials and methods; [Burge et al., 2016]). In other words, the model is ‘image-computable’: the model computes the image cues from image pixels and produces tilt estimates as outputs.

We approximate the posterior mean ${\displaystyle E\left[\tau |\mathbf{C}\right]=\sum _{\tau }\tau p\left(\tau |\mathbf{C}\right)}$ by computing the sample mean of the groundtruth tilt conditional on each unique image cue triplet (Figure 5A). The result is a table, or ‘estimate cube,’ where each cell stores the optimal estimate ${\displaystyle {\hat{\tau }}_{MMSE}=E\left[\tau |\mathbf{C}\right]}$ for a particular combination of image cues (Figure 5B).

Figure 5 with 1 supplement see all

Download asset Open asset

In the cue-combination literature, cues are commonly assumed to be statistically independent (Ernst and Banks, 2002). In natural scenes, it is not clear whether this assumption holds. Fortunately, the normative model used here is free of assumptions about statistical independence and the form of the joint probability distribution (see Discussion). Thus, our normative model provides a principled benchmark, grounded in natural scene statistics, against which to compare human performance.

We tested the model observer on the exact same set of natural stimuli used to test human observers (Figure 5C). The model observer predicts the overall pattern of raw human responses (see also Figure 5—figure supplement 1). More impressively, the model observer predicts the counts, means, and variances of the human tilt estimates (Figure 2D–F), the conditional error distributions (Figure 3), and the conditional groundtruth tilt distributions (Figure 4). The model explains a large proportion of the variance for all of these performance measures (Figure 5D). These results indicate that human visual system estimates tilt in accordance with optimal processes that minimize error in natural scenes. We conclude that the biased and imprecise human tilt estimates with natural stimuli are nevertheless lawful.

Two points are worth emphasizing. First, this model observer had no free parameters that were fit to the human data (Burge et al., 2016); instead, the model observer was designed to perform the task optimally given the three image cues. Second, the close agreement between human and model performance suggests that humans use the same cues (or cues that strongly correlate with those) used by the normative model (see Discussion).

Trial-by-trial error

If human and model observers use the same cues in natural stimuli to estimate tilt, variation in the stimuli should cause similar variation in performance. Are human performance and model observer performance similar in individual trials? The same set of natural stimuli was presented to all observers. Thus, it is possible to make direct, trial-by-trial comparisons of the estimation errors that each observer made. If the properties of individual natural stimuli influence estimates similarly across observers, then observer errors across trials should be correlated. Accounting for trial-by-trial errors is one of the most stringent comparisons that can be made between model and human performance.

Natural stimuli do elicit similar trial-by-trial errors from human and model observers (Figure 6A). The model predicts trial-by-trial human errors far better than chance. We quantify the model-human similarity with the circular correlation coefficients of the trial-by-trial model and human estimates (Figure 6B). The correlation coefficients are significant. This result implies that the errors are systematically and reliably dependent on the properties of natural stimuli and that these properties affect human and model observers similarly.

Figure 6 with 2 supplements see all

Download asset Open asset

However, because both human and model observers produced biased estimates with natural stimuli (Figure 2E, Figure 2—figure supplement 2), it is possible that the biases are responsible for the error correlations. To remove the possible influence of bias, we computed the bias-corrected error. On each trial, we subtracted the observer bias at each groundtruth tilt ${\displaystyle {\mathbf{e}}^{\ast }=\stackrel{error}{\overbrace{\left(\hat{\tau }-\tau \right)}}-E(\stackrel{bias}{\overbrace{\hat{\tau }-\tau |\tau }})}$ from the raw error. Human and model bias-corrected errors are also significantly correlated (Figure 6C,D). The human-human correlation (dashed line in Figure 6B,D; see Figure 6—figure supplement 1) sets an upper bound for the model-human correlation. The model-human correlation approaches this bound in some cases. Other measures of trial-by-trial similarity (e.g., choice probability; Figure 6—figure supplement 2C) yield similar conclusions. These results show that natural stimulus variation at a given groundtruth tilt causes similar response variation in human observers and the model observer.

To ensure that the predictive power of the model observer is not trivial, we developed multiple alternative models. All other models predict human performance more poorly (Figure 6—figure supplement 2). Our results do not rule out the possibility that another model could predict human performance better, but the current MMSE estimator establishes a strong benchmark against which other models must be compared.

Thus, the normative model, without fitting to the human data, accounts for human tilt estimates at the level of the summary statistics (Figure 2D–F), the conditional distributions (Figure 3 and Figure 4), and the trial-by-trial errors (Figure 6). Together, this evidence suggests that the human visual system’s perceptual processes and the normative model’s computations are making similar use of similar information. We conclude that the human visual system makes near-optimal use of the available information in natural stimuli for estimating 3D surface tilt.

Performance-impacting stimulus factors: Slant, distance, and natural depth variation

In our experiment, natural and artificial stimuli were matched on many dimensions: tilt, slant, distance, and luminance contrast. These stimulus factors are commonly controlled in perceptual experiments. Consistent with previous reports, slant and distance had a substantial impact on estimation error (Watt et al., 2005) with both natural and artificial stimuli (Figure 7). (Luminance contrast had little impact on performance.)

Figure 7

Download asset Open asset

Even after controlling for these stimulus dimensions, tilt estimation with natural stimuli is considerably poorer than tilt estimation with artificial stimuli. Other factors must therefore account for the differences. What are they? In our experiment, each artificial scene consisted of a single planar surface. Natural scenes contain natural depth variation (i.e., complex surface structure); some surfaces are approximately planar, some are curved or bumpy. How are differences in surface planarity related to differences in performance with natural and artificial scenes? To quantify the departure of surface structure from planarity, we defined local tilt variance as the circular variance of the groundtruth tilt values in the central 1° area of each stimulus. Then, we examined how estimation error changes with tilt variance.

First, we found that estimation error increases linearly with tilt variance for both human and model observers (Figure 8A). Unfortunately, tilt variance co-varies with groundtruth tilt — cardinal tilts tend to have lower tilt variance than oblique tilts, presumably because of the ground plane (Figure 8B)— which means that the effect of groundtruth tilt could be misattributed to tilt variance. Hence, we repeated the analysis of overall error separately for cardinal tilts alone and for oblique tilts alone. We found that the effect of tilt variance is independent of groundtruth tilt (Figure 8C). Thus, like slant and distance, tilt variance (i.e., departure from surface planarity) is one of several key stimulus factors that impacts tilt estimation performance.

Figure 8 with 1 supplement see all

Download asset Open asset

Second, we found that for near-planar natural stimuli, average estimation error with natural and artificial stimuli are closely matched (left-most points in Figure 8A). Does this result mean that tilt variance accounts for all performance differences between natural and artificial stimuli? No. Performance with near-planar natural stimuli is still substantially different from performance with artificial stimuli (Figure 8—figure supplement 1). In addition, individual human and model trial-by-trial estimation errors are still correlated for the near-planar natural stimuli. Furthermore, the patterns of human performance with natural stimuli are robust across a wide range of tilt variance. Figure 9 shows the summary statistics (estimate counts, means, and variances; cf. Figure 2D–F) for multiple different tilt variances of human observers. Model performance is also similarly robust to tilt variance (Figure 9—figure supplement 1).

Figure 9 with 1 supplement see all

Download asset Open asset

We conclude that although tilt variance is an important performance-modulating factor, it is not the only factor responsible for performance differences with natural and artificial stimuli. Other factors must be responsible. Understanding these other factors is an important direction for future work.

Estimating 3D surface orientation requires the estimation of both slant and tilt. The current study focuses on tilt estimation. We quantify performance in natural scenes and report that human tilt estimates are often neither accurate nor precise. To connect our work to the classic literature, we matched artificial and natural stimuli on the stimulus dimensions that are controlled most often in typical experiments. The comparison revealed systematic performance differences. The detailed patterns of human performance are predicted, without free parameters to fit the data, by a normative model that is grounded in natural scene statistics and that makes the best possible use of the available image information. Importantly, this model is distinguished from many models of mid-level visual tasks because it is ‘image computable’; that is, it takes image pixels as input and produces tilt estimates as output. Together, the current experiment and modeling effort contributes to a broad goal in vision and visual neuroscience research: to generalize our understanding of human vision from the lab to the real world.

1. 1. Adams WJ
   2. Elder JH
   3. Graf EW
   4. Leyland J
   5. Lugtigheid AJ
   6. Muryy A

   (2016) The Southampton-York Natural Scenes (SYNS) dataset: Statistics of surface attitude

   Scientific Reports 6:35805.

   https://doi.org/10.1038/srep35805

   • PubMed
   • Google Scholar
2. 1. Appelle S

   (1972) Perception and discrimination as a function of stimulus orientation: the "oblique effect" in man and animals

   Psychological Bulletin 78:266–278.

   https://doi.org/10.1037/h0033117

   • PubMed
   • Google Scholar
3. 1. Banks MS
   2. Gepshtein S
   3. Landy MS

   (2004) Why is spatial stereoresolution so low?

   Journal of Neuroscience 24:2077–2089.

   https://doi.org/10.1523/JNEUROSCI.3852-02.2004

   • PubMed
   • Google Scholar
4. 1. Burge J
   2. Fowlkes CC
   3. Banks MS

   (2010b) Natural-scene statistics predict how the figure-ground cue of convexity affects human depth perception

   Journal of Neuroscience 30:7269–7280.

   https://doi.org/10.1523/JNEUROSCI.5551-09.2010

   • PubMed
   • Google Scholar
5. 1. Burge J
   2. Geisler WS

   (2011) Optimal defocus estimation in individual natural images

   PNAS 108:16849–16854.

   https://doi.org/10.1073/pnas.1108491108

   • PubMed
   • Google Scholar
6. Conference

   1. Burge J
   2. Geisler WS

   (2012) Optimal defocus estimates from individual images for autofocusing a digital camera

   Proceedings of SPIE.

   https://doi.org/10.1117/12.912066

   • Google Scholar
7. 1. Burge J
   2. Geisler WS

   (2014) Optimal disparity estimation in natural stereo images

   Journal of Vision 14:1.

   https://doi.org/10.1167/14.2.1

   • PubMed
   • Google Scholar
8. 1. Burge J
   2. Geisler WS

   (2015) Optimal speed estimation in natural image movies predicts human performance

   Nature Communications 6:7900.

   https://doi.org/10.1038/ncomms8900

   • PubMed
   • Google Scholar
9. 1. Burge J
   2. Girshick AR
   3. Banks MS

   (2010a) Visual-haptic adaptation is determined by relative reliability

   Journal of Neuroscience 30:7714–7721.

   https://doi.org/10.1523/JNEUROSCI.6427-09.2010

   • PubMed
   • Google Scholar
10. 1. Burge J
    2. Jaini P

    (2017) Accuracy maximization analysis for sensory-perceptual tasks: Computational improvements, filter robustness, and coding advantages for scaled additive noise

    PLoS Computational Biology 13(2):e1005281.

    https://doi.org/10.1371/journal.pcbi.1005281

    • PubMed
    • Google Scholar
11. 1. Burge J
    2. McCann BC
    3. Geisler WS

    (2016) Estimating 3D tilt from local image cues in natural scenes

    Journal of Vision 16:2.

    https://doi.org/10.1167/16.13.2

    • PubMed
    • Google Scholar
12. 1. Burgess AE
    2. Wagner RF
    3. Jennings RJ
    4. Barlow HB

    (1981) Efficiency of human visual signal discrimination

    Science 214:93–94.

    https://doi.org/10.1126/science.7280685

    • PubMed
    • Google Scholar
13. 1. Clerc M
    2. Mallat S

    (2002) The texture gradient equation for recovering shape from texture

    IEEE Transactions on Pattern Analysis and Machine Intelligence 24:536–549.

    https://doi.org/10.1109/34.993560

    • Google Scholar
14. 1. Dorr M
    2. Martinetz T
    3. Gegenfurtner KR
    4. Barth E

    (2010) Variability of eye movements when viewing dynamic natural scenes

    Journal of Vision 10:28.

    https://doi.org/10.1167/10.10.28

    • PubMed
    • Google Scholar
15. 1. Ernst MO
    2. Banks MS

    (2002) Humans integrate visual and haptic information in a statistically optimal fashion

    Nature 415:429–433.

    https://doi.org/10.1038/415429a

    • PubMed
    • Google Scholar
16. 1. Fleming RW
    2. Holtmann-Rice D
    3. Bülthoff HH

    (2011) Estimation of 3D shape from image orientations

    PNAS 108:20438–20443.

    https://doi.org/10.1073/pnas.1114619109

    • PubMed
    • Google Scholar
17. 1. Furmanski CS
    2. Engel SA

    (2000) An oblique effect in human primary visual cortex

    Nature Neuroscience 3:535–536.

    https://doi.org/10.1038/75702

    • PubMed
    • Google Scholar
18. Conference

    1. Galasso F
    2. Lasenby J

    (2007) Shape from texture: fast estimation of planar surface orientation via fourier analysis

    Procedings of the British Machine Vision Conference 2007.

    https://doi.org/10.5244/C.21.71

    • Google Scholar
19. 1. Geisler WS
    2. Najemnik J
    3. Ing AD

    (2009) Optimal stimulus encoders for natural tasks

    Journal of Vision 9:17.1–16.

    https://doi.org/10.1167/9.13.17

    • PubMed
    • Google Scholar
20. 1. Girshick AR
    2. Landy MS
    3. Simoncelli EP

    (2011) Cardinal rules: visual orientation perception reflects knowledge of environmental statistics

    Nature Neuroscience 14:926–932.

    https://doi.org/10.1038/nn.2831

    • PubMed
    • Google Scholar
21. 1. Hillis JM
    2. Watt SJ
    3. Landy MS
    4. Banks MS

    (2004) Slant from texture and disparity cues: optimal cue combination

    Journal of Vision 4:1–92.

    https://doi.org/10.1167/4.12.1

    • PubMed
    • Google Scholar
22. 1. Jaini P
    2. Burge J

    (2017) Linking normative models of natural tasks to descriptive models of neural response

    Journal of Vision 17(12):16:1–26.

    https://doi.org/10.1167/17.12.16

    • Google Scholar
23. Book

    1. Knill DC
    2. Richards W

    (1996) Perception as Bayesian Inference

    New York: Cambridge University Press.

    https://doi.org/10.1017/CBO9780511984037

    • Google Scholar
24. 1. Knill DC

    (1998a) Surface orientation from texture: ideal observers, generic observers and the information content of texture cues

    Vision Research 38:1655–1682.

    https://doi.org/10.1016/S0042-6989(97)00324-6

    • PubMed
    • Google Scholar
25. 1. Knill DC

    (1998b) Ideal observer perturbation analysis reveals human strategies for inferring surface orientation from texture

    Vision Research 38:2635–2656.

    https://doi.org/10.1016/S0042-6989(97)00415-X

    • PubMed
    • Google Scholar
26. 1. Land MF
    2. Hayhoe M

    (2001) In what ways do eye movements contribute to everyday activities?

    Vision Research 41:3559–3565.

    https://doi.org/10.1016/S0042-6989(01)00102-X

    • PubMed
    • Google Scholar
27. 1. Li A
    2. Zaidi Q

    (2000) Perception of three-dimensional shape from texture is based on patterns of oriented energy

    Vision Research 40:217–242.

    https://doi.org/10.1016/S0042-6989(99)00169-8

    • PubMed
    • Google Scholar
28. 1. Li A
    2. Zaidi Q

    (2004) Three-dimensional shape from non-homogeneous textures: carved and stretched surfaces

    Journal of Vision 4:3–78.

    https://doi.org/10.1167/4.10.3

    • PubMed
    • Google Scholar
29. 1. Malik J
    2. Rosenholtz R

    (1997) Computing local surface orientation and shape from texture for curved surfaces

    International Journal of Computer Vision 23:149–168.

    https://doi.org/10.1023/A:1007958829620

    • Google Scholar
30. 1. Marlow PJ
    2. Todorović D
    3. Anderson BL

    (2015) Coupled computations of three-dimensional shape and material

    Current Biology 25:R221–R222.

    https://doi.org/10.1016/j.cub.2015.01.062

    • PubMed
    • Google Scholar
31. Book

    1. Marr D

    (1982)

    Vision: A Computational Investigation into the Human Representation and Processing of Visual Information

    New York: W H Freeman & Company.

    • Google Scholar
32. 1. Massot C
    2. Hérault J

    (2008) Model of frequency analysis in the visual cortex and the shape from texture problem

    International Journal of Computer Vision 76:165–182.

    https://doi.org/10.1007/s11263-007-0048-x

    • Google Scholar
33. 1. McDermott J

    (2004) Psychophysics with junctions in real images

    Perception 33:1101–1127.

    https://doi.org/10.1068/p5265

    • PubMed
    • Google Scholar
34. 1. Murphy AP
    2. Ban H
    3. Welchman AE

    (2013) Integration of texture and disparity cues to surface slant in dorsal visual cortex

    Journal of Neurophysiology 110:190–203.

    https://doi.org/10.1152/jn.01055.2012

    • PubMed
    • Google Scholar
35. 1. Norman JF
    2. Todd JT
    3. Norman HF
    4. Clayton AM
    5. McBride TR

    (2006) Visual discrimination of local surface structure: slant, tilt, and curvedness

    Vision Research 46:1057–1069.

    https://doi.org/10.1016/j.visres.2005.09.034

    • PubMed
    • Google Scholar
36. 1. Oruç I
    2. Maloney LT
    3. Landy MS

    (2003) Weighted linear cue combination with possibly correlated error

    Vision Research 43:2451–2468.

    https://doi.org/10.1016/S0042-6989(03)00435-8

    • PubMed
    • Google Scholar
37. 1. Pelz JB
    2. Rothkopf C

    (2007)

    Eye Movements: A Window on Mind and Brain

    Oculomotor behavior in natural and man-made environments, Eye Movements: A Window on Mind and Brain, 10.1016/B978-008044980-7/50033-1.

    • Google Scholar
38. 1. Rosenberg A
    2. Cowan NJ
    3. Angelaki DE

    (2013) The visual representation of 3D object orientation in parietal cortex

    Journal of Neuroscience 33:19352–19361.

    https://doi.org/10.1523/JNEUROSCI.3174-13.2013

    • PubMed
    • Google Scholar
39. 1. Rosenholtz R
    2. Malik J

    (1997) Surface orientation from texture: isotropy or homogeneity (or both)?

    Vision Research 37:2283–2293.

    https://doi.org/10.1016/S0042-6989(96)00121-6

    • PubMed
    • Google Scholar
40. 1. Sanada TM
    2. Nguyenkim JD
    3. Deangelis GC

    (2012) Representation of 3-D surface orientation by velocity and disparity gradient cues in area MT

    Journal of Neurophysiology 107:2109–2122.

    https://doi.org/10.1152/jn.00578.2011

    • PubMed
    • Google Scholar
41. 1. Saunders JA
    2. Knill DC

    (2001) Perception of 3D surface orientation from skew symmetry

    Vision Research 41:3163–3183.

    https://doi.org/10.1016/S0042-6989(01)00187-0

    • PubMed
    • Google Scholar
42. 1. Stevens KA

    (1983) Slant-tilt: the visual encoding of surface orientation

    Biological Cybernetics 46:183–195.

    https://doi.org/10.1007/BF00336800

    • PubMed
    • Google Scholar
43. 1. Stocker AA
    2. Simoncelli EP

    (2006) Noise characteristics and prior expectations in human visual speed perception

    Nature Neuroscience 9:578–585.

    https://doi.org/10.1038/nn1669

    • PubMed
    • Google Scholar
44. 1. Todd JT
    2. Koenderink JJ
    3. van Doorn AJ
    4. Kappers AM

    (1996) Effects of changing viewing conditions on the perceived structure of smoothly curved surfaces

    Journal of Experimental Psychology: Human Perception and Performance 22:695–706.

    https://doi.org/10.1037/0096-1523.22.3.695

    • PubMed
    • Google Scholar
45. 1. Todd JT

    (2004) The visual perception of 3D shape

    Trends in Cognitive Sciences 8:115–121.

    https://doi.org/10.1016/j.tics.2004.01.006

    • PubMed
    • Google Scholar
46. 1. Tsutsui K
    2. Jiang M
    3. Yara K
    4. Sakata H
    5. Taira M

    (2001) Integration of perspective and disparity cues in surface-orientation-selective neurons of area CIP

    Journal of Neurophysiology 86:2856–2867.

    https://doi.org/10.1152/jn.2001.86.6.2856

    • PubMed
    • Google Scholar
47. 1. Tyler CW
    2. Julesz B

    (1978) Binocular cross-correlation in time and space

    Vision Research 18:101–105.

    https://doi.org/10.1016/0042-6989(78)90083-4

    • PubMed
    • Google Scholar
48. 1. Velisavljević L
    2. Elder JH

    (2006) Texture properties affecting the accuracy of surface attitude judgements

    Vision Research 46:2166–2191.

    https://doi.org/10.1016/j.visres.2006.01.010

    • PubMed
    • Google Scholar
49. 1. Watt SJ
    2. Akeley K
    3. Ernst MO
    4. Banks MS

    (2005) Focus cues affect perceived depth

    Journal of Vision 5:7–62.

    https://doi.org/10.1167/5.10.7

    • PubMed
    • Google Scholar
50. 1. Weiss Y
    2. Simoncelli EP
    3. Adelson EH

    (2002) Motion illusions as optimal percepts

    Nature Neuroscience 5:598–604.

    https://doi.org/10.1038/nn0602-858

    • PubMed
    • Google Scholar
51. 1. Welchman AE
    2. Deubelius A
    3. Conrad V
    4. Bülthoff HH
    5. Kourtzi Z

    (2005) 3D shape perception from combined depth cues in human visual cortex

    Nature Neuroscience 8:820–827.

    https://doi.org/10.1038/nn1461

    • PubMed
    • Google Scholar
52. 1. Yang Z
    2. Purves D

    (2003a) A statistical explanation of visual space

    Nature Neuroscience 6:632–640.

    https://doi.org/10.1038/nn1059

    • PubMed
    • Google Scholar
53. 1. Yang Z
    2. Purves D

    (2003b) Image/source statistics of surfaces in natural scenes

    Network: Computation in Neural Systems 14:371–390.

    https://doi.org/10.1088/0954-898X_14_3_301

    • PubMed
    • Google Scholar

Author details

1. Seha Kim

   Department of Psychology, University of Pennsylvania, Philadelphia, United States

   Contribution

   Conceptualization, Resources, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   For correspondence

   sehakim@upenn.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-0356-6168

2. Johannes Burge

   Department of Psychology, University of Pennsylvania, Philadelphia, United States

   Contribution

   Conceptualization, Resources, Software, Formal analysis, Supervision, Funding acquisition, Validation, Investigation, Visualization, Methodology, Writing—original draft, Project administration, Writing—review and editing

   For correspondence

   jburge@sas.upenn.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-0311-7875

• 1,183

  views

• 116

  downloads

• 22

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.