Neuroanatomical structures within the medial temporal lobe (MTL) are known to support memory-related behaviors (Scoville and Milner, 1957; Eichenbaum and Cohen, 2004; LaRocque and Wagner, 2015). For decades, experimentalists have also observed MTL-related impairments in tasks designed to test perceptual processing (Suzuki, 2009; Baxter, 2009). These findings centered on perirhinal cortex (PRC), an MTL structure situated at the apex of high-level sensory cortices (Figure 1a). Visual impairments were reported following lesions to PRC in humans and other animals, bolstering a perceptual-mnemonic account of perirhinal function (e.g. Murray and Bussey, 1999; Bussey et al., 2002; Lee et al., 2005; Lee et al., 2006; Barense et al., 2007; Inhoff et al., 2019). However, there were also visual experiments for which no impairments were observed following PRC lesions (e.g. Buffalo et al., 1998a; Buffalo et al., 1998b; Stark and Squire, 2000; Knutson et al., 2012). In this was, decades of evidence resulted in a pattern of seemingly inconsistent experimental outcomes, with no formal method for disambiguating between competing interpretations of the available data.

Figure 1

Download asset Open asset

One of the central challenges in this experimental literature has been isolating PRC-dependent behaviors from those supported by PRC-adjacent sensory cortex. In the primate, this requires disentangling PRC-dependent performance from visual behaviors supported by the ventral visual stream (VVS; DiCarlo and Cox, 2007; DiCarlo et al., 2012). Lacking more objective metrics, experimentalists had relied on informal, descriptive accounts of perceptual demands; terms such as ‘complexity’ and ‘feature ambiguity’ were intended to characterize those stimulus properties that are necessary to evaluate PRC involvement in visual object perception. However, this informal approach led to conflicting interpretations of the available evidence, without any means to arbitrate between them. For example, the absence of PRC-related deficits in a given study (e.g. Stark and Squire, 2000) has led to the conclusion that PRC is not involved in perception (Suzuki, 2009), while others argue that stimuli in stimuli from these studies are not ‘complex’ enough (i.e. can be represented by canonical visual cortices) and so no perceptual deficits are expected (Bussey and Saksida, 2002).

In recent years, deep learning computational methods have become commonplace in the vision sciences. Remarkably, these models are able to predict neural responses throughout the primate VVS directly from experimental stimuli: given an experimental image as input, these models (e.g. convolutional neural networks, CNNs) are able to predict neural responses. These ‘stimulus-comptable’ methods currently provide the most quantitatively accurate predictions of neural responses throughout the primate VVS (Yamins et al., 2014; Khaligh-Razavi and Kriegeskorte, 2014; Rajalingham et al., 2018; Bashivan et al., 2019). For example, early model layers within a CNN better predict earlier stages of processing within the VVS (e.g. V4; Figure 1b: left, gray) while later model layers better predict later stages of processing within the VVS (e.g. IT; Figure 1b: left, green). We note that there is not a 1–1 correspondence between these models and the primate VVS as they typically lack known biological properties (Zhuang et al., 2021; Doerig et al., 2022). Nonetheless, these models can be modified to evaluate domain-specific hypotheses (Doerig et al., 2022)—for example by adding recurrence (Kubilius et al., 2018; Kietzmann et al., 2019) or eccentricity-dependent scaling (Deza and Konkle, 2020; Jonnalagadda et al., 2021).

Recently, Bonnen et al., 2021 leveraged these ‘VVS-like’ models to evaluate the performance of PRC-intact/-lesioned human participants in visual discrimination tasks. While VVS-like models are able to approximate performance supported by a linear readout of high-level visual cortex (Figure 1b: right, green), human participants are able to out outperform both VVS-like models and a linear readout of direct electrophysiological recordings from the VVS (Figure 1b: right, purple). Critically, VVS-like models approximate PRC-lesioned performance. While these data implicate PRC in visual object processing, there remains experimental data collected from non-human primates which have not been formally evaluated. Like the human literature, non-human primate data have been used to both support and refute PRC involvement in perception. Unlike the naturally occurring lesioned in humans, experiments with non-human primates have unparalleled control over the site and extent of PRC lesions—potentially, providing more incisive tests of competing claims over PRC function. As such, characterizing the discrepancies between human and non-human primate data is a critical step toward developing a more formal understanding of PRC involvement in perception.

In order to resolve this cross-species discrepancy, here we formalize perceptual demands in experiments administered to PRC-intact/-lesioned monkeys (Macaca mulatta). We draw from data collected by Eldridge et al., 2018 which provides striking evidence against PRC involvement in perception: Eldridge et al., 2018 created multiple stimulus sets, allowing for more a fine-grained evaluation of perceptual behaviors than previous, related work (e.g. Bussey et al., 2003). Here, we estimate VVS-supported performance on stimuli from Eldridge et al., 2018 and compare these predictions to PRC-intact and -lesioned choice behaviors. This modeling approach enables us to situate human and macaque lesion data within a shared metric space (i.e. VVS-model performance); as such, previous observations in the human (e.g. Figure 1b: right, green) constrain how data from Eldridge et al., 2018 can be interpreted; critically, to evaluate PRC involvement in perception, the performance of non-lesioned participants must exceed VVS-modeled performance. Given this, supra-VVS performance may be due to PRC-dependent contributions (schematized in Figure 1c: left), or for reasons unrelated to PRC function (schematized in Figure 1c: middle). However, if VVS-supported performance approximates PRC-intact behavior, no perceptual processing beyond the VVS should be necessary (schematized in Figure 1c: right). We refer to stimuli in this category as ‘non-diagnostic’.

We begin with a task-optimized convolutional neural network, pretrained to perform object classification. We estimate the correspondence between this model and electrophysiological responses from high-level visual cortex using a protocol previously reported in Bonnen et al., 2021. We summarize this protocol here, but refer to the previous manuscript for a more detailed account. Using previously collected electrophysiological responses from macaque VVS (Majaj et al., 2015), we identify a model layer that best fits high-level visual cortex: Given a set of images, we learn a linear mapping between model responses and a single electrode’s responses, then evaluate this mapping using independent data (i.e. left-out images). For each model layer, this analysis yields a median cross-validated fit to noise-corrected neural responses, for both V4 and IT. As is consistently reported (e.g. Schrimpf et al., 2020), early model layers (i.e. first half of layers) better predict neural responses in V4 than do later layers (unpaired t-test: ${\displaystyle t(8)=2.70,\mathrm{P}=0.015}$; Figure 1b: left, gray), while later layers better predict neural responses in IT, a higher-level region (unpaired t-test: ${\displaystyle t(8)=3.70,\mathrm{P}=0.002}$; Figure 1b: left, green). Peak V4 fits occur in model layer pool3 (noise-corrected ${\displaystyle \mathrm{r}=0.95\pm 0.30}$ STD) while peak IT fits occur in con5_1 (noise-corrected ${\displaystyle \mathrm{r}=0.88\pm 0.16}$ STD). For ease, in all subsequent analyses we use model responses from a con5_1-adjacent layer, fc6, which has comparable neural fits but a lower-dimensional representation.

Next we compare model, VVS-supported, and human performance within the same metric space: Instead of fitting model responses directly to electrophysiological recordings in high-level visual cortex, as above, here we evaluate the similarity between the performance supported by the model and high-level visual cortex, as well as human performance on these same stimuli. For this comparison, we leverage electrophysiological responses previously collected from macaque IT cortex (Majaj et al., 2015), using a protocol originally detailed in Bonnen et al., 2021. We independently estimate model and VVS-supported performance on a stimulus set composed of concurrent visual discrimination trials, using a modified leave-one-out cross-validation strategy. We then determine the model-VVS fit over the performance estimates, as developed in Bonnen et al., 2021 and outlined in Methods. We can compare model performance with both VVS-supported performance and PRC-intact (human, n = 297) performance on these same stimuli, using data from Bonnen et al., 2021. On this dataset, a computational proxy for the VVS predicts IT-supported performance (${\displaystyle \beta =0.81}$, ${\displaystyle \mathrm{F}(1,30)=13.33}$, ${\displaystyle \mathrm{P}=4\times {10}^{-14}}$; Figure 1b, green), while each are outperformed by (${\displaystyle \beta =0.24}$, $t(31)=9.50$, ${\displaystyle \mathrm{P}=1\times {10}^{-10}}$; Figure 1b: right, purple). These data suggest that while these models are suitable proxies for VVS-supported performance, human performance is able to exceed a linear readout of the VVS.

With these ‘VVS-like’ models, we turn to analyses of macaque lesion data. First, we extract model responses to each stimulus in all four experiments administered by Eldridge et al., 2018. In these experiments, subjects provided a binary classification for each stimulus: ‘cat’ or ‘dog.’ Critically, stimuli were composed not only of cats and dogs, but of ‘morphed’ images that parametrically vary the percent of category-relevant information present in each trial. For example, ‘10% morphs’ were 90% cat and 10% dog. These morphed stimuli were designed to evaluate PRC involvement in perception by creating maximal ‘feature ambiguity,’ a perceptual quality reported to elicit PRC dependence in previous work (Bussey et al., 2002; Norman and Eacott, 2004; Bussey et al., 2006; Murray and Richmond, 2001). On each trial, subjects were rewarded for responses that correctly identify which category best fits the image presented (e.g. 10% = ‘cat’, 90% = ‘dog’, correct response is ‘dog’). We evaluate data from two groups of monkeys in this study: an unoperated control group (n = 3) and a group with bilateral removal of rhinal cortex, which including peri- and entorhinal cortex. We formulate the modeling problem as a binary forced choice (i.e. ‘dog’ = 1, ‘cat’ = 0) and present the model with experimental stimuli. We then extract model responses from a layer that corresponds to ‘high-level’ visual cortex and learn a linear mapping from model responses to predict the category label. For all analyses, we report the results on held-out data (Methods: Determining model performance).

We first evaluate model performance with the aggregate metrics used by the original authors—not on the performance of individual images, but on the proportion of trials within the same ‘bin’ that are correct. With the original behavioral data, we average performance across images within each morph level (e.g. 10%, 20%, etc.) across subjects in each lesion group (PRC-intact Figure 2a, and -lesioned Figure 2b). As reported in Eldridge et al., 2018, there is not a significant difference between the choice behaviors of PRC-lesioned and -intact subjects (no significant difference between PRC-intact/-lesion groups: $R^2=0.00$, $\beta =0.01$, $F(1,86)$ = ${\displaystyle 0.07,\mathrm{P}=0.941}$). For each of these experiments, we extract model responses to all stimuli from a model layer that best corresponds to a high-level visual region, inferior temporal (IT) cortex. Using the model responses from this ‘IT-like’ model layer to each image, we train a linear, binary classification model on the category label of each image (i.e. ‘dog’ or ‘cat’) on 4/5th of the available stimuli. We then evaluate model performance on the remaining 1/5th of those stimuli, repeating this procedure across 50 iterations of randomized train–test splits. A computational proxy for the VVS exhibits the same qualitative pattern of behavior as each subject group (Figure 2c, model performance across multiple train–test iterations in black). Moreover, we observe a striking correspondence between model and PRC-intact behavior (Figure 2b, purple: $R^2=0.98$, $\beta =0.97$, ${\displaystyle t(21)=33.12,\mathrm{P}=6\times {10}^{-19}}$ ) as well as -lesioned subjects (green: $R^2=0.99$, $\beta =0.96$, ${\displaystyle t(21)=57.38,\mathrm{P}=1\times {10}^{-23}}$). Employing the same metric used to claim no significant difference between PRC-lesion/-intact performance, we find no difference between subject and model behavior ($R^2=0.00$, $\beta =-0.01$, ${\displaystyle F(1,86)=-0.11,\mathrm{P}=0.915}$).

Figure 2 with 4 supplements see all

Download asset Open asset

We extend our analysis beyond the aggregate morph- and subject-level analyses used by the original authors, introducing a split-half reliability analysis (Methods: Split-half reliability estimates). This enables us to determine if there is reliable choice behavior, for each subject, at the level of individual images. We restrict our analyses to experiments with sufficient data, as this analysis requires multiple repetitions of each image; we exclude experiments 3 (’Masked Morphs’) and 4 (’Crossed Morphs’) due to insufficient repetitions (which can be seen in Figure 2, rows 3–4). Across both remaining experiments, we find consistent image-level choice behaviors for subjects with an intact (e.g. median ${\displaystyle R_{\text{exp1}}^2=0.94}$, median ${\displaystyle R_{\text{exp2}}^2=0.86}$) and lesioned (e.g. median ${\displaystyle R_{\text{exp1}}^2=0.91}$, median ${\displaystyle R_{\text{exp2}}^2=0.90}$) rhinal cortex (Figure 3a: within-subject reliability on the diagonal; PRC-intact subjects in purple, PRC-lesioned subjects in green). We also observe consistent image-level choice behaviors between subjects (e.g. median ${\displaystyle R_{\text{exp1}}^2=0.86}$, median ${\displaystyle R_{\text{exp2}}^2=0.79}$). These results indicate there is reliable within- and between-subject variance in the image-by-image choice behaviors of experimental subjects (Figure 3a: PRC-intact subjects in purple, PRC-lesioned subjects in green; between-group reliability in gray), suggesting that this behavior is a suitable target to evaluate how well we approximate more granular subject behaviors with a computational proxy for the VVS. We next examine whether the model can predict these more granular, subject- and image-level choice behaviors (see Methods: Consistency estimates).

Figure 3 with 1 supplement see all

Download asset Open asset

Our computational approach is able to predict subject-level choice behavior when aggregated across morph levels, for both PRC-intact (e.g. subject 0; $R^2=0.99\beta =1.01$, ${\displaystyle t(21)=39.30,\mathrm{P}=2\times {10}^{-20}}$) and -lesioned (e.g. subject 4: $R^2=0.99\beta =1.01$, ${\displaystyle t(21)=45.01,\mathrm{P}=1\times {10}^{-21}}$) subjects (Figure 3b). Interestingly, the model’s fit to subject behavior is indistinguishable from the distribution of between-subject reliability estimates (Figure 1a; median of the empirical p(model ${\displaystyle |{\text{reliability}}_{\text{between-subject}})=0.592}$) suggesting that the model exhibits ‘subject-like’ behaviors at this resolution. Our modeling approach is also able to significantly predict image-level choice behaviors for both PRC-lesioned (e.g. subject 3: $R^2=0.86\beta =0.81$, ${\displaystyle \mathrm{F}(1,438)=52.79,\mathrm{P}=5\times {10}^{-192}}$) and -intact subjects (e.g. subject 1: $R^2=0.87\beta =0.88$, ${\displaystyle \mathrm{F}(1,438)=53.24,\mathrm{P}=2\times {10}^{-193}}$). However, the model behavior is unlikely to be observed under the distribution of between-subject reliability estimates (between-subject reliability distributions visualized in Figure 1b; median of the empirical p(model $|{\text{reliability}}_{\text{between-subject}})=0$). That is, the model does not exhibit ‘subject-like’ choice behaviors at the resolution of individual images. This is an important caveat to note when evaluating the correspondence between model performance and animal behavior: as previously reported (Rajalingham et al., 2018), even as these models approximate neural responses and choice behaviors in the aggregate (i.e. across images), they do not necessarily capture the trial-by-trial choice behaviors. We elaborate on this further in the discussion.

There are properties of the experimental design in Eldridge et al., 2018 that encourage a more careful comparison between primate and model behavior. Experimental stimuli contain discrete interpolations between ‘cat’ and ‘dog’ images, such that adjacent stimuli within a morph sequence are highly similar (e.g. see Figure 2—figure supplement 1). The colinearity in this stimulus set is revealed by running a classification analysis over pixels: a linear readout of stimulus category directly from the vectorized (i.e. flattened) images themselves is sufficient to approximate aggregate performance of all experimental groups ($R^2=0.94\beta =0.90$, ${\displaystyle \mathrm{F}(1,42)=26.74}$ , ${\displaystyle \mathrm{P}=5\times {10}^{-28}}$ ; Figure 2—figure supplement 2). To ensure that the VVS modeling approach is not simply a byproduct of the colinearity in the stimuli, we construct a conservative method for model evaluation by restricting training data to images from unrelated morphs sequences (i.e. train on morph sequences A–F, test on morph sequence G). Under this more conservative train–test split, pixels are no longer predictive of primate behavior ($R^2=0.05\beta =0.45$, ${\displaystyle \mathrm{F}(1,42)=1.42}$ , ${\displaystyle \mathrm{P}=0.164}$; Figure 2—figure supplement 3), but there remains a clear correspondence between the model and PRC-lesioned ($R^2=0.87\beta =1.17$, ${\displaystyle \mathrm{F}(1,42)=16.94}$ , ${\displaystyle \mathrm{P}=2\times {10}^{-20}}$ ) and -intact performance ($R^2=0.88\beta =1.16$, ${\displaystyle \mathrm{F}(1,42)=17.39,\mathrm{P}=8\times {10}^{-21}}$; Figure 2—figure supplement 4). That is, although subjects were able to exploit the colinearity in the stimuli to improve their performance with experience, the correspondence between VVS models and primate choice behaviors is not an artifact of these low-level stimulus attributes.

There are properties of the experimental design in Eldridge et al., 2018 that encourage a more careful comparison between primate and model behavior. Experimental stimuli contain discrete interpolations between ‘cat’ and ‘dog’ images, such that adjacent stimuli within a morph sequence are highly similar (e.g. see Figure 2—figure supplement 1). The colinearity in this stimulus set is revealed by running a classification analysis over pixels: a linear readout of stimulus category directly from the vectorized (i.e. flattened) images themselves is sufficient to approximate aggregate performance of all experimental groups ($R^2=0.94\beta =0.90$, ${\displaystyle \mathrm{F}(1,42)=26.74}$ , ${\displaystyle \mathrm{P}=5\times {10}^{-28}}$ ; Figure 2—figure supplement 2). To ensure that the VVS modeling approach is not simply a byproduct of the colinearity in the stimuli, we construct a conservative method for model evaluation by restricting training data to images from unrelated morphs sequences (i.e. train on morph sequences A–F, test on morph sequence G). Under this more conservative train–test split, pixels are no longer predictive of primate behavior ($R^2=0.05\beta =0.45$, ${\displaystyle \mathrm{F}(1,42)=1.42}$ , ${\displaystyle \mathrm{P}=0.164}$; Figure 2—figure supplement 3), but there remains a clear correspondence between the model and PRC-lesioned ($R^2=0.87\beta =1.17$, ${\displaystyle \mathrm{F}(1,42)=16.94}$ , ${\displaystyle \mathrm{P}=2\times {10}^{-20}}$ ) and -intact performance ($R^2=0.88\beta =1.16$, ${\displaystyle \mathrm{F}(1,42)=17.39,\mathrm{P}=8\times {10}^{-21}}$; Figure 2—figure supplement 4). That is, although subjects were able to exploit the colinearity in the stimuli to improve their performance with experience, the correspondence between VVS models and primate choice behaviors is not an artifact of these low-level stimulus attributes.

To evaluate competing claims surrounding PRC involvement in perception, Eldridge et al., 2018 administered a series of visual classification tasks to PRC-lesioned/-intact monkeys. These stimuli were carefully crafted to exhibit a qualitative, perceptual property that had previously been shown to elicit PRC dependence (i.e. ‘feature ambiguity’; Bussey et al., 2002; Norman and Eacott, 2004; Bussey et al., 2006; Murray and Richmond, 2001). The absence of PRC-related deficits across four experiments led the original authors to suggest that perceptual processing is not dependent on PRC. Here, we reevaluate this claim by situating these results within a more formal computational framework; leveraging task-optimized convolutional neural networks as a proxy for primate visual processing (Yamins et al., 2014; Rajalingham et al., 2018; Schrimpf et al., 2020). We first determined VVS-model performance on the experimental stimuli in Eldridge et al., 2018. We then compared these computational results with monkey choice behaviors, including subjects with bilateral lesions to PRC (n = 3), as well as unoperated controls (n = 3). For both PRC-lesioned/-intact monkeys, we observe a striking correspondence between VVS model and experimental behavior at the group (Figure 2d) and subject level (Figure 3b). These results suggest that a linear readout of the VVS should be sufficient to enable the visual classification behaviors in Eldridge et al., 2018; no PRC-related impairments are expected.

In isolation, it is ambiguous how these data should be interpreted. For example, if VVS-modeled accuracy was sufficient to explain PRC-intact performance across all known stimulus sets, this would suggest that PRC is not involved in visual object perception. However, previous computational results from humans demonstrate that PRC-intact participants are able to outperform a linear readout of the VVS (schematized in Figure 1b: right, purple). Because results from these human experiments are in the same metric space as our current results (i.e. VVS-modeled performance), these data unambiguously constrain our interpretation: for a stimulus set to evaluate PRC involvement in visual processing, participants must be able to outperform a linear readout of the VVS. That is, supra-VVS performance must be observed in order to isolate PRC contributions from those of other possible contributors to these behaviors (e.g. prefrontal cortex, schematized in Figure 1c: center). Given that supra-VVS performance is not observed in the current stimulus set (Figure 2d; schematized in Figure 1c: right), we conclude that experiments in Eldridge et al., 2018 are not diagnostic of PRC involvement in perception. Consequently, we suggest that these data do not offer absolute evidence against PRC involvement in perception—revising the original conclusions made from this study.

We note that there is meaningful variance in the trial-level behaviors not captured by the current modeling framework. By conducting a more granular analyses than the original study (i.e. an image-level analysis, instead of averaging across multiple images within the same morph level), we found that image-level choice behaviors are reliable both within and between subjects (Figure 3a). At this image-level resolution, however, the VVS model does not match the pattern of choice behaviors evident in experimental subjects (Figure 3c Figure 3—figure supplement 1). This observation is consistent with previous reports (Rajalingham et al., 2018), suggesting that these VVS-like models are best suited to approximate aggregate choice behaviors, not responses to individual images. Many sources of variance have been identified as possible contributors to these subject–model divergences, such as biologically implausible training data (Zhuang et al., 2021), or lack of known properties of the primate visual system—for example recurrence (Kar and DiCarlo, 2020) or eccentricity-dependent scaling (Jonnalagadda et al., 2021).

While admittedly coarse, these computational proxies for the VVS provide an unprecedented opportunity to understand perirhinal function. Their contribution is, principally, to isolate PRC-dependent behaviors from those supported by the VVS. More generally, however, this is possible because these methods directly interface with experimental data—making predictions of VVS-supported performance directly from experimental stimuli, instead of relying on the discretion of experimentalists. This stimulus-computable property of these models provides a formal ‘linking function’ between theoretical claims with experimental evidence. In turn, this modeling approach creates a unified metric space (in this case, ‘model performance’) that enables us to evaluate experimental outcomes across labs, across studies, and even across species. We believe that a judicious application of these computational tools, alongside a careful consideration of animal behavior, will enrich the next generation of empirical studies surrounding MTL-dependent perceptual processing.

All scripts used for analysis and visualization can be accessed via github at https://github.com/tzler/eldridge_reanalysis (copy archived at Bonnen, 2023). All stimuli and behavioral data used in these analyses can be downloaded via Dryad at https://doi.org/10.5061/dryad.r4xgxd2h7.

1. 1. Bonnen T
   2. Eldridge M

   (2022) Dryad Digital Repository

   Data from: Inconsistencies between human and macaque lesion data can be resolved with a stimulus-computable model of the ventral visual stream.

   https://doi.org/10.5061/dryad.r4xgxd2h7

1. 1. Barense MD
   2. Gaffan D
   3. Graham KS

   (2007) The human medial temporal lobe processes Online representations of complex objects

   Neuropsychologia 45:2963–2974.

   https://doi.org/10.1016/j.neuropsychologia.2007.05.023

   • PubMed
   • Google Scholar
2. 1. Bashivan P
   2. Kar K
   3. DiCarlo JJ

   (2019) Neural population control via deep image synthesis

   Science 364:eaav9436.

   https://doi.org/10.1126/science.aav9436

   • PubMed
   • Google Scholar
3. 1. Baxter MG

   (2009) Involvement of medial temporal lobe structures in memory and perception

   Neuron 61:667–677.

   https://doi.org/10.1016/j.neuron.2009.02.007

   • PubMed
   • Google Scholar
4. 1. Bonnen T
   2. Yamins DLK
   3. Wagner AD

   (2021) When the ventral visual stream is not enough: A deep learning account of medial temporal lobe involvement in perception

   Neuron 109:2755–2766.

   https://doi.org/10.1016/j.neuron.2021.06.018

   • PubMed
   • Google Scholar
5. Software

   1. Bonnen T

   (2023) Analysis and visualization scripts, version swh:1:rev:f04b8ab7a44e297bc94e9ef77c3eb20f80d16b91

   Software Heritage.

   https://archive.softwareheritage.org/swh:1:dir:af4e8e55fbe804078f631cca8abb816b0d563819;origin=https://github.com/tzler/eldridge_reanalysis;visit=swh:1:snp:be1cd4e1f66f55f6b82e97a28e5112cb987fa079;anchor=swh:1:rev:f04b8ab7a44e297bc94e9ef77c3eb20f80d16b91
6. 1. Buffalo EA
   2. Reber PJ
   3. Squire LR

   (1998a) The human Perirhinal cortex and recognition memory

   Hippocampus 8:330–339.

   https://doi.org/10.1002/(SICI)1098-1063(1998)8:4<330::AID-HIPO3>3.0.CO;2-L

   • PubMed
   • Google Scholar
7. 1. Buffalo EA
   2. Stefanacci L
   3. Squire LR
   4. Zola SM

   (1998b) A reexamination of the concurrent discrimination learning task: the importance of anterior Inferotemporal cortex, area Te

   Behavioral Neuroscience 112:3–14.

   https://doi.org/10.1037//0735-7044.112.1.3

   • PubMed
   • Google Scholar
8. 1. Bussey TJ
   2. Saksida LM

   (2002) The organization of visual object representations: a Connectionist model of effects of lesions in Perirhinal cortex

   European Journal of Neuroscience 15:355–364.

   https://doi.org/10.1046/j.0953-816x.2001.01850.x

   • Google Scholar
9. 1. Bussey TJ
   2. Saksida LM
   3. Murray EA

   (2002) Perirhinal cortex resolves feature ambiguity in complex visual discriminations

   European Journal of Neuroscience 15:365–374.

   https://doi.org/10.1046/j.0953-816x.2001.01851.x

   • Google Scholar
10. 1. Bussey TJ
    2. Saksida LM
    3. Murray EA

    (2003) Impairments in visual discrimination after Perirhinal cortex lesions: testing ‘declarative’ vs. ‘perceptual-Mnemonic’ views of Perirhinal cortex function

    The European Journal of Neuroscience 17:649–660.

    https://doi.org/10.1046/j.1460-9568.2003.02475.x

    • PubMed
    • Google Scholar
11. 1. Bussey TJ
    2. Saksida LM
    3. Murray EA

    (2006) Perirhinal cortex and feature-ambiguous discriminations

    Learning & Memory 13:103–105.

    https://doi.org/10.1101/lm.163606

    • Google Scholar
12. Conference

    1. Deng J
    2. Dong W
    3. Socher R
    4. Li LJ

    (2009) ImageNet: A large-scale hierarchical image database

    2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops). pp. 248–255.

    https://doi.org/10.1109/CVPR.2009.5206848

    • Google Scholar
13. Preprint

    1. Deza A
    2. Konkle T

    (2020) Emergent Properties of Foveated Perceptual Systems

    arXiv.

    https://arxiv.org/abs/2006.07991

    • Google Scholar
14. 1. DiCarlo JJ
    2. Cox DD

    (2007) Untangling invariant object recognition

    Trends in Cognitive Sciences 11:333–341.

    https://doi.org/10.1016/j.tics.2007.06.010

    • Google Scholar
15. 1. DiCarlo JJ
    2. Zoccolan D
    3. Rust NC

    (2012) How does the brain solve visual object recognition?

    Neuron 73:415–434.

    https://doi.org/10.1016/j.neuron.2012.01.010

    • PubMed
    • Google Scholar
16. Preprint

    1. Doerig A
    2. Sommers R
    3. Seeliger K

    (2022) The Neuroconnectionist Research Programme

    arXiv.

    https://arxiv.org/abs/2209.03718

    • Google Scholar
17. Book

    1. Eichenbaum H
    2. Cohen NJ

    (2004) From Conditioning to Conscious Recollection: Memory Systems of the Brain

    Oxford University Press on Demand.

    https://doi.org/10.1093/acprof:oso/9780195178043.001.0001

    • Google Scholar
18. 1. Eldridge MA
    2. Matsumoto N
    3. Wittig JH
    4. Masseau EC
    5. Saunders RC
    6. Richmond BJ

    (2018) Perceptual processing in the ventral visual stream requires area Te but not Rhinal cortex

    eLife 7:e36310.

    https://doi.org/10.7554/eLife.36310

    • PubMed
    • Google Scholar
19. 1. Inhoff MC
    2. Heusser AC
    3. Tambini A
    4. Martin CB
    5. O’Neil EB
    6. Köhler S
    7. Meager MR
    8. Blackmon K
    9. Vazquez B
    10. Devinsky O
    11. Davachi L

    (2019) Understanding Perirhinal contributions to perception and memory: evidence through the lens of selective Perirhinal damage

    Neuropsychologia 124:9–18.

    https://doi.org/10.1016/j.neuropsychologia.2018.12.020

    • PubMed
    • Google Scholar
20. Preprint

    1. Jonnalagadda A
    2. Wang WY
    3. Manjunath B
    4. Eckstein MP

    (2021) Foveater: Foveated Transformer for Image Classification

    arXiv.

    https://arxiv.org/abs/2105.14173

    • Google Scholar
21. Preprint

    1. Kar K
    2. DiCarlo JJ

    (2020) Fast Recurrent Processing via Ventral Prefrontal Cortex Is Needed by the Primate Ventral Stream for Robust Core Visual Object Recognition

    bioRxiv.

    https://doi.org/10.1101/2020.05.10.086959

    • Google Scholar
22. 1. Khaligh-Razavi SM
    2. Kriegeskorte N

    (2014) Deep supervised, but not Unsupervised, models may explain IT cortical representation

    PLOS Computational Biology 10:e1003915.

    https://doi.org/10.1371/journal.pcbi.1003915

    • PubMed
    • Google Scholar
23. 1. Kietzmann TC
    2. Spoerer CJ
    3. Sörensen LKA
    4. Cichy RM
    5. Hauk O
    6. Kriegeskorte N

    (2019) Recurrence is required to capture the representational Dynamics of the human visual system

    PNAS 116:21854–21863.

    https://doi.org/10.1073/pnas.1905544116

    • PubMed
    • Google Scholar
24. 1. Knutson AR
    2. Hopkins RO
    3. Squire LR

    (2012) Visual discrimination performance, memory, and medial temporal lobe function

    PNAS 109:13106–13111.

    https://doi.org/10.1073/pnas.1208876109

    • PubMed
    • Google Scholar
25. Preprint

    1. Kubilius J
    2. Schrimpf M
    3. Nayebi A
    4. Bear D
    5. Yamins DLK
    6. DiCarlo JJ

    (2018) CORnet: Modeling the Neural Mechanisms of Core Object Recognition

    bioRxiv.

    https://doi.org/10.1101/408385

    • Google Scholar
26. Book

    1. LaRocque K
    2. Wagner AD

    (2015)

    The Medial Temporal Lobe and Episodic Memory

    Elsevier.

    • Google Scholar
27. 1. Lee ACH
    2. Bussey TJ
    3. Murray EA
    4. Saksida LM
    5. Epstein RA
    6. Kapur N
    7. Hodges JR
    8. Graham KS

    (2005) Perceptual deficits in amnesia: challenging the medial temporal lobe ‘Mnemonic’View

    Neuropsychologia 43:1–11.

    https://doi.org/10.1016/j.neuropsychologia.2004.07.017

    • Google Scholar
28. 1. Lee ACH
    2. Buckley MJ
    3. Gaffan D
    4. Emery T
    5. Hodges JR
    6. Graham KS

    (2006) Differentiating the roles of the hippocampus and Perirhinal cortex in processes beyond long-term declarative memory: a double dissociation in dementia

    The Journal of Neuroscience 26:5198–5203.

    https://doi.org/10.1523/JNEUROSCI.3157-05.2006

    • Google Scholar
29. 1. Majaj NJ
    2. Hong H
    3. Solomon EA
    4. DiCarlo JJ

    (2015) Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance

    The Journal of Neuroscience 35:13402–13418.

    https://doi.org/10.1523/JNEUROSCI.5181-14.2015

    • PubMed
    • Google Scholar
30. 1. Murray EA
    2. Bussey TJ

    (1999) Perceptual–Mnemonic functions of the Perirhinal cortex

    Trends in Cognitive Sciences 3:142–151.

    https://doi.org/10.1016/S1364-6613(99)01303-0

    • Google Scholar
31. 1. Murray EA
    2. Richmond BJ

    (2001) Role of Perirhinal cortex in object perception, memory, and associations

    Current Opinion in Neurobiology 11:188–193.

    https://doi.org/10.1016/s0959-4388(00)00195-1

    • PubMed
    • Google Scholar
32. 1. Norman G
    2. Eacott MJ

    (2004) Impaired object recognition with increasing levels of feature ambiguity in rats with Perirhinal cortex lesions

    Behavioural Brain Research 148:79–91.

    https://doi.org/10.1016/s0166-4328(03)00176-1

    • PubMed
    • Google Scholar
33. 1. Pedregosa F
    2. Varoquaux G
    3. Gramfort A

    (2011)

    Scikit-learn: machine learning in python

    Journal of Machine Learning Research 12:2825–2830.

    • Google Scholar
34. 1. Rajalingham R
    2. Issa EB
    3. Bashivan P
    4. Kar K
    5. Schmidt K
    6. DiCarlo JJ

    (2018) Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks

    The Journal of Neuroscience 38:7255–7269.

    https://doi.org/10.1523/JNEUROSCI.0388-18.2018

    • PubMed
    • Google Scholar
35. 1. Schrimpf M
    2. Kubilius J
    3. Lee MJ
    4. Ratan Murty NA
    5. Ajemian R
    6. DiCarlo JJ

    (2020) Integrative Benchmarking to advance Neurally mechanistic models of human intelligence

    Neuron 108:413–423.

    https://doi.org/10.1016/j.neuron.2020.07.040

    • PubMed
    • Google Scholar
36. 1. Scoville WB
    2. Milner B

    (1957) Loss of recent memory after bilateral hippocampal lesions

    Journal of Neurology, Neurosurgery, and Psychiatry 20:11–21.

    https://doi.org/10.1136/jnnp.20.1.11

    • PubMed
    • Google Scholar
37. Preprint

    1. Simonyan K
    2. Zisserman A

    (2014) Very Deep Convolutional Networks for Large-Scale Image Recognition

    arXiv.

    https://arxiv.org/abs/1409.1556

    • Google Scholar
38. 1. Stark CEL
    2. Squire LR

    (2000) Intact visual perceptual discrimination in humans in the absence of Perirhinal cortex

    Learning & Memory 7:273–278.

    https://doi.org/10.1101/lm.35000

    • Google Scholar
39. 1. Suzuki WA

    (2009) Perception and the medial temporal lobe: evaluating the current evidence

    Neuron 61:657–666.

    https://doi.org/10.1016/j.neuron.2009.02.008

    • PubMed
    • Google Scholar
40. 1. Yamins DLK
    2. Hong H
    3. Cadieu CF
    4. Solomon EA
    5. Seibert D
    6. DiCarlo JJ

    (2014) Performance-Optimized Hierarchical models predict neural responses in higher visual cortex

    PNAS 111:8619–8624.

    https://doi.org/10.1073/pnas.1403112111

    • PubMed
    • Google Scholar
41. 1. Zhuang C
    2. Yan S
    3. Nayebi A
    4. Schrimpf M
    5. Frank MC
    6. DiCarlo JJ
    7. Yamins DLK

    (2021) Unsupervised neural network models of the ventral visual stream

    PNAS 118:e2014196118.

    https://doi.org/10.1073/pnas.2014196118

    • PubMed
    • Google Scholar

Decision letter after peer review:

[Editors’ note: the authors submitted for reconsideration following the decision after peer review. What follows is the decision letter after the first round of review.]

Thank you for submitting your Research Advance "Inconsistencies between human and macaque lesion data can be resolved with a stimulus-computable model of the ventral visual stream" for consideration by eLife. Your article has been reviewed by 2 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and a Senior Editor. The following individual involved in the review of your submission has agreed to reveal their identity: Jonathan Winawer (Reviewer #2).

Reviewer #1 (Recommendations for the authors):

This article describes the application of a computational model, previously published in 2021 in Neuron, to an empirical dataset from monkeys, previously published in 2018 in eLife. The 2021 modeling paper argued that the model can be used to determine whether a particular task depends on the perirhinal cortex as opposed to being soluble using ventral visual stream structures alone. The 2018 empirical paper used a series of visual discrimination tasks in monkeys that were designed to contain high levels of 'feature ambiguity' (in which the stimuli that must be discriminated share a large proportion of overlapping features), and yet animals with rhinal cortex lesions were unimpaired, leading the authors to conclude that perirhinal cortex is not involved in the visual perception of objects. The present article revisits and revises that conclusion: when the 2018 tasks are run through the 2021 computational model, the model suggests that they should not depend on perirhinal cortex function after all, because the model of VVS function achieves the same levels of performance as both controls and PRC-lesioned animals from the 2018 paper. This leads the authors of the present study to conclude that the 2018 data are simply "non-diagnostic" in terms of the involvement of the perirhinal cortex in object perception.

The authors have successfully applied the computational tool from 2021 to empirical data, in exactly the way the tool was designed to be used. To the extent that the model can be accepted as a veridical proxy for primate VVS function, its conclusions can be trusted and this study provides a useful piece of information in the interpretation of often contradictory literature. However, I found the contribution to be rather modest. The results of this computational study pertain to only a single empirical study from the literature on perirhinal function (Eldridge et al, 2018). Thus, it cannot be argued that by reinterpreting this study, the current contribution resolves all controversy or even most of the controversy in the foregoing literature. The Bonnen et al. 2021 paper provided a potentially useful computational tool for evaluating the empirical literature, but using that tool to evaluate (and ultimately rule out as non-diagnostic) a single study does not seem to warrant an entire manuscript: I would expect to see a reevaluation of a much larger sample of data in order to make a significant contribution to the literature, above and beyond the paper already published in 2021. In addition, the manuscript in its current form leaves the motivations for some analyses under-specified and the methods occasionally obscure.

– The manuscript does not make a compelling argument as to why Eldridge et al. (2018) is a particularly important example of the prior literature whose reevaluation will change the interpretation of the literature as a whole.

– Considerable effort is expended on evaluating how well the model can "approximate more granular subject behaviors" but it is not explained why this is important, or whether it matters that the model cannot, in fact, approximate image-level subject behavior.

– The section "determining model performance" does not provide sufficient detail for a reader to reproduce the modeling work. The statement that "we estimate the optimal regularization strength for the logistic regression model" appears to be the only statement detailing how the model is trained. This is too sparse and opaque and needs expanding considerably.

– The section "8.2 Consistency estimates" and the caption to Figure S4 both refer to the procedure for estimating the correspondence between subject-subject or subject-model choice behaviors. But these two sections appear to contradict each other. The figure caption says that the authors generate a random split of each subject's data. But in Section 8.2, the last sentence implies (although it's not completely clear) that for the between-subjects metric, all the data from each subject is used. (And it is true that, for a between-subjects analysis, you could use all the data to compute a correlation). Please clarify exactly how the 'split' was generated and whether a split was used for all analyses including between subjects.

Reviewer #2 (Recommendations for the authors):

The goal of this paper is to use a model-based approach, developed by one of the authors and colleagues in 2021, to critically re-evaluate the claims made in a prior paper from 2018, written by the other author of this paper (and colleagues), concerning the role of perirhinal cortex in visual perception. The prior paper compared monkeys with and without lesions to the perirhinal cortex and found that their performance was indistinguishable on a difficult perceptual task (categorizing dog-cat morphs as dogs or cats). Because the performance was the same, the conclusion was that the perirhinal cortex is not needed for this task, and probably not needed for perception in general, since this task was chosen specifically to be a task that the perirhinal cortex *might* be important for. Well, the current work argues that in fact the task and stimuli were poorly chosen since the task can be accomplished by a model of the ventral visual cortex. More generally, the authors start with the logic that the perirhinal cortex gets input from the ventral visual processing stream and that if a task can be performed by the ventral visual processing stream alone, then the perirhinal cortex will add no benefit to that task. Hence to determine whether the perirhinal cortex plays a role in perception, one needs a task (and stimulus set) that cannot be done by the ventral visual cortex alone (or cannot be done at the level of monkeys or humans).

There are two important questions the authors then address. First, can their model of the ventral visual cortex perform as well as macaques (with no lesion) on this task? The answer is yes, based on the analysis of this paper. The second question is, are there any tasks that humans or monkeys can perform better than their ventral visual model? If not, then maybe the ventral visual model (and biological ventral visual processing stream) is sufficient for all recognition. The answer here too is yes, there are some tasks humans can perform better than the model. These then would be good tasks to test with a lesion approach to the perirhinal cortex. It is worth noting, though, that none of the analyses showing that humans can outperform the ventral visual model are included in this paper - the papers which showed this are cited but not discussed in detail.

Major strength:

The computational and conceptual frameworks are very valuable. The authors make a compelling case that when patients (or animals) with perirhinal lesions perform equally to those without lesions, the interpretation is ambiguous: it could be that the perirhinal cortex doesn't matter for perception in general, or it could be that it doesn't matter for this stimulus set. They now have a way to distinguish these two possibilities, at least insofar as one trusts their ventral visual model (a standard convolutional neural network). While of course, the model cannot be perfectly accurate, it is nonetheless helpful to have a concrete tool to make a first-pass reasonable guess at how to disambiguate results. Here, the authors offer a potential way forward by trying to identify the kinds of stimuli that will vs won't rely on processing beyond the ventral visual stream. The re-interpretation of the 2018 paper is pretty compelling.

Major weakness:

It is not clear that an off-the-shelf convolution neural network really is a great model of the ventral visual stream. Among other things, it lacks eccentricity-dependent scaling. It also lacks recurrence (as far as I could tell). To the authors' credit, they show detailed analysis on an image-by-image basis showing that in fine detail the model is not a good approximation of monkey choice behavior. This imposes limits on how much trust one should put in model performance as a predictor of whether the ventral visual cortex is sufficient to do a task or not. For example, suppose the authors had found that their model did more poorly than the monkeys (lesioned or not lesioned). According to their own logic, they would have, it seems, been led to the interpretation that some area outside of the ventral visual cortex (but not the perirhinal cortex) contributes to perception, when in fact it could have simply been that their model missed important aspects of ventral visual processing. That didn't happen in this paper, but it is a possible limitation of the method if one wanted to generalize it. There is work suggesting that recurrence in neural networks is essential for capturing the pattern of human behavior on some difficult perceptual judgments (e.g., Kietzmann et al 2019, PNAS). In other words, if the ventral model does not match human (or macaque) performance on some recognition task, it does not imply that an area outside the ventral stream is needed - it could just be that a better ventral model (eg with recurrence, or some other property not included in the model) is needed. This weakness pertains to the generalizability of the approach, not to the specific claims made in this paper, which appear sound.

A second issue is that the title of the paper, "Inconsistencies between human and macaque lesion data can be resolved with a stimulus-computable model of the ventral visual stream" does not seem to be supported by the paper. The paper challenges a conclusion about macaque lesion data. What inconsistency is reconciled, and how?

[Editors’ note: The authors appealed the original decision. What follows is the authors’ response to the first round of review.]

Reviewer #1 (Recommendations for the authors):

This article describes the application of a computational model, previously published in 2021 in Neuron, to an empirical dataset from monkeys, previously published in 2018 in eLife. The 2021 modeling paper argued that the model can be used to determine whether a particular task depends on the perirhinal cortex as opposed to being soluble using ventral visual stream structures alone. The 2018 empirical paper used a series of visual discrimination tasks in monkeys that were designed to contain high levels of 'feature ambiguity' (in which the stimuli that must be discriminated share a large proportion of overlapping features), and yet animals with rhinal cortex lesions were unimpaired, leading the authors to conclude that perirhinal cortex is not involved in the visual perception of objects. The present article revisits and revises that conclusion: when the 2018 tasks are run through the 2021 computational model, the model suggests that they should not depend on perirhinal cortex function after all, because the model of VVS function achieves the same levels of performance as both controls and PRC-lesioned animals from the 2018 paper. This leads the authors of the present study to conclude that the 2018 data are simply "non-diagnostic" in terms of the involvement of the perirhinal cortex in object perception.

We appreciate the Reviewer’s careful reading and synthesis of the background and general findings of this manuscript.

The authors have successfully applied the computational tool from 2021 to empirical data, in exactly the way the tool was designed to be used. To the extent that the model can be accepted as a veridical proxy for primate VVS function, its conclusions can be trusted and this study provides a useful piece of information in the interpretation of often contradictory literature. However, I found the contribution to be rather modest. The results of this computational study pertain to only a single empirical study from the literature on perirhinal function (Eldridge et al, 2018). Thus, it cannot be argued that by reinterpreting this study, the current contribution resolves all controversy or even most of the controversy in the foregoing literature. The Bonnen et al. 2021 paper provided a potentially useful computational tool for evaluating the empirical literature, but using that tool to evaluate (and ultimately rule out as non-diagnostic) a single study does not seem to warrant an entire manuscript: I would expect to see a reevaluation of a much larger sample of data in order to make a significant contribution to the literature, above and beyond the paper already published in 2021. In addition, the manuscript in its current form leaves the motivations for some analyses under-specified and the methods occasionally obscure.

We believe that our comments below outline our rationale for focusing our current analysis on data from Eldridge et al. In brief, these data provide compelling evidence against PRC involvement in perception, and are the only such data with PRC-lesioned/-intact macaques that we were able to secure the stimuli for. As such, data from Eldridge et al. provide a singular opportunity to address discrepancies between human and macaque lesion data. For this reason, we propose the current work as a Research Advance Article type, building off of a manuscript that was previously published in eLife.

– The manuscript does not make a compelling argument as to why Eldridge et al. (2018) is a particularly important example of the prior literature whose reevaluation will change the interpretation of the literature as a whole.

As the Reviewer implies, there are multiple visual discrimination experiments administered to PRC-lesioned and -intact monkeys. We offer two reasons here why we have chosen to focus our analyses on Eldridge et al. 2018.

First, Eldridge et al. 2018 is currently the only relevant visual discrimination experiment administered to PRC-lesioned/-intact macaques for which we have been able to secure experimental stimuli. Prior to the submitting the current manuscript, we solicited authors of the following studies used to support and refute PRC involvement in visual perception:

• Buffalo, E. A., Ramus, S. J., Clark, R. E., Teng, E., Squire, L. R., & Zola, S. M. (1999). Dissociation between the effects of damage to perirhinal cortex and area TE. Learning & Memory, 6(6), 572-599.

• Buckley, M. J., Booth, M. C., Rolls, E. T., & Gaffan, D. (2001). Selective perceptual impairments after perirhinal cortex ablation. Journal of Neuroscience, 21(24), 9824-9836.

• Bussey, T. J., Saksida, L. M., & Murray, E. A. (2002). Perirhinal cortex resolves feature ambiguity in complex visual discriminations. European Journal of Neuroscience, 15(2), 365-374.

• Bussey, T. J., Saksida, L. M., & Murray, E. A. (2006). Perirhinal cortex and feature-ambiguous discriminations. Learning & Memory, 13(2), 103-105

• Bussey, T. J., Saksida, L. M., & Murray, E. A. (2003). Impairments in visual discrimination after perirhinal cortex lesions: testing ‘declarative’ vs. ‘perceptual‐mnemonic’ views of perirhinal cortex function. European Journal of Neuroscience, 17(3), 649-660.

• Eldridge, M. A., Matsumoto, N., Wittig, J. H., Masseau, E. C., Saunders, R. C., & Richmond, B. J. (2018). Perceptual processing in the ventral visual stream requires area TE but not rhinal cortex. elife, 7, e36310.

After reaching out to the original authors, we were only able to secure stimuli from Buckley et al. 2001 and Eldridge et al. 2018. However, the Buckley et al. stimuli have previously been modeled/published by Bonnen, Yamins, and Wagner, 2021 (identical stimuli were later used in human PRC lesioned experiments, with consistent results). As such, we believe that modeling stimuli from Eldridge et al. provide the only novel contribution to the field.

Second, we believe that findings from Eldridge et al. provide the most incisive test of the cross-species discrepancies between human and macaque lesion studies. Currently, the human lesion data have been shown to be entirely consistent PRC involvement in visual object perception (Bonnen, Yamins, and Wagner, 2021). Unlike other macaque stimulus sets we have access to (i.e. Buckley et al. 2001) data from Eldridge et al. were used to refute accounts of PRC involvement in perception. As such, the relative impact of accounting for data in Eldridge et al. is far greater than the majority of studies present in the literature. Moreover, because the design used in Eldridge et al. is different from those used in the human lesion experiments previously modeled (including Buckley et al. 2001) this experiment provides a powerful proof of principle that this modeling framework is useful to understand PRC involvement in perception not only across species, but also across experimental designs.

– Considerable effort is expended on evaluating how well the model can "approximate more granular subject behaviors" but it is not explained why this is important, or whether it matters that the model cannot, in fact, approximate image-level subject behavior.

In order to highlight the logic to these analyses, we have provided a brief explanation in the Results section and clarified our ideas in the discussion. In sum, we hope that this modeling approach will be useful to future experimentalists, and so would like to make clear what the limitations are on predicting animal behaviors. That is, this analysis is not directly relevant to claims about perirhinal function, but more of a methodological claim about model abilities.

From the Results section:

“Our computational approach is able to predict subject-level choice behavior when aggregated across morph levels, for both PRC-intact (e.g. subject 0; R2 = 0.99 β = 1.01, t(21) = 39.30, P = 2 x 10−20) and -lesioned (e.g. subject 4: R2 = 0.99 β = 1.01, t(21) = 45.01, P = 1 x 10−21) subjects (Figure 3b). Interestingly, the model’s fit to subject behavior is indistinguishable from the distribution of between-subject reliability estimates (Figure 1a; median of the empirical P(model|reliability_{between-subject}) = 0.592) suggesting that the model exhibits ‘subject-like’ behaviors at this resolution. Our modeling approach is also able to significantly predict image-level choice behaviors for both PRC-lesioned (e.g. subject 3: R2 = 0.86 β = 0.81, F(1, 438) = 52.79, P = 5 x 10−192) and -intact subjects (e.g. subject 1: R2 = 0.87 β = 0.88, F(1, 438) = 53.24, P = 2 x 10−193). However, the model behavior is unlikely to be observed under the distribution of between-subject reliability estimates (between-subject reliability distributions visualized in Figure 1b; median of the empirical P(model|reliability_{between-subject}) = 0). That is, the model does not exhibit ‘subject-like’ choice behaviors at the resolution of individual images. This is an important caveat to note when evaluating the correspondence between model performance and animal behavior: as previously reported (Rajalingham et al., 2018), even as these models approximate neural responses and choice behaviors in the aggregate (i.e. across images), they do not necessarily capture the trial-by-trial choice behaviors. We elaborate on this further in the discussion.”

From the Discussion section:

“We note that there is meaningful variance in the trial-level behaviors not captured by the current modeling framework. By conducting a more granular analyses than the original study (i.e. an image-level analysis, instead of averaging across multiple images within the same morph level), we found that image-level choice behaviors are reliable both within- and between-subjects (Figure 3a). At this image-level resolution, however, the VVS model does not match the pattern of choice behaviors evident in experimental subjects (Figure 3c; Supplemental Figure 1b). This observation is consistent with previous reports (Rajalingham et al., 2018), suggesting that these VVS-like models are best suited to approximate aggregate choice behaviors, not responses to individual images. Many sources of variance have been identified as possible contributors to these subject-model divergences, such as biologically implausible training data (Zhuang et al., 2021), or lack of known properties of the primate visual system—e.g. recurrence (Kar and DiCarlo, 2020) or eccentricity-dependent scaling (Jonnalagadda et al., 2021).”

– The section "determining model performance" does not provide sufficient detail for a reader to reproduce the modeling work. The statement that "we estimate the optimal regularization strength for the logistic regression model" appears to be the only statement detailing how the model is trained. This is too sparse and opaque and needs expanding considerably.

We appreciate this request for clarification and agree that this section was not sufficiently clear. We have expanded our description in this section as outlined below:

“For all estimates of model performance we use a task-optimized convolutional neural network pretrained on Imagenet (Deng et al., 2009). For transparency, we report the results from the first instance of this model class used to evaluate these data (Simonyan and Zisserman, 2014), but note that these results hold across all model instances evaluated. We preprocess each image from Eldridge et al., 2018 using a standard computer vision preprocessing pipeline; resizing images to a width and height of 224x224, then normalizing each image by the mean ([0.485, 0.456, 0.406]) and standard deviation ([0.229, 0.224, 0.225]) of the distribution of images used to train this model. We present each preprocessed image to the model and extract responses to each image from a layer (fc6) that exhibits a high correspondence with electrophysiological responses to high-level visual cortex (Bonnen et al., 2021; and see Figure 1b: left). For each experiment, we generate a random train-test split, using 4/5th of the data to train a linear readout (in this case, a logistic regression model). To train this linear readout from model responses, we use a l2-normed logistic regression model implemented in sklearn (Pedregosa et al., 2011) to predict the binary category classification (i.e. ‘dog’=1, ‘cat’=0) for each image in the training set. Within the training set, we estimate the optimal regularization strength (‘C’ from 10−5 to 10−5) for the logistic regression model through 5-fold cross validation. We then evaluate model performance on each experiment on independent data (i.e. the remaining 1/5th of stimuli). We repeat this process for 100 permutations (i.e. random 4/5th splits) of stimuli in each condition. Each iteration’s model predictions (on independent data) are plotted in Figure 2c.”

– The section "8.2 Consistency estimates" and the caption to Figure S4 both refer to the procedure for estimating the correspondence between subject-subject or subject-model choice behaviors. But these two sections appear to contradict each other. The figure caption says that the authors generate a random split of each subject's data. But in Section 8.2, the last sentence implies (although it's not completely clear) that for the between-subjects metric, all the data from each subject is used. (And it is true that, for a between-subjects analysis, you could use all the data to compute a correlation). Please clarify exactly how the 'split' was generated and whether a split was used for all analyses including between subjects.

We appreciate this request for clarification and agree that this section was not sufficiently clear. We have added a few sentences to the “consistency estimates” section in order to clarify that the splits for the within- and between-subject analyses are generated in an identical manner, as well as including some of the rationale behind this decision:

“We estimate within- and between-subject consistency using a common protocol. For the given resolution of analysis (either morph- or image-level), we require multiple presentations of the same items. For the morph-level analysis, which aggregates stimuli within ‘morph levels’ (e.g. all stimuli that are designed to be 0% dog, 10%, etc.), all stimulus sets meet this criterion. There are, however, multiple experiments that do not contain sufficient data to perform the image-level analysis, which requires multiple presentations of each stimulus; experiment four contains only one presentation of each stimulus, precluding it from our consistency analyses, and experiment three contains only 4 repetitions, which is insufficient for reliable within- and between-subject consistency estimates. Thus, we restrict our consistency estimates to experiments one (10 repetitions per image) and two (8 repetitions per image).

We estimate all consistency metrics over 100 iterations of random split-halves. For each iteration, across all items within a given resolution (where items can refer to either a given morph percent, for the morph-level analysis, or a given image, for the image-level analysis), we randomly split choice behavior into two random splits. In the image-level analysis, for example, for each image x_i within the set of n images, we randomly select half of all trials of x_i (i.e.${\displaystyle x_{i_1}}$), and compute the mean of this random sample (${\displaystyle x_{i_1}}$). We repeate this for all of the n images in this condition (i.e. generating ${\displaystyle {\overline{x}}_{1_1,}{\overline{x}}_{2_1}...,{\overline{x}}_{n_1}}$). We repeat this procedure for the remaining half of trial on each n images (i.e. generating ${\displaystyle {\overline{x}}_{1_2,}{\overline{x}}_{2_2},...,{\overline{x}}_{n_2}}$). Thus, we have two n dimensional vectors, ${\displaystyle {\overrightarrow{v}}_1}$ and ${\displaystyle {\overrightarrow{v}}_2}$, where the element in each vector corresponds to a random half of trials drawn from all trial containing that image. We use R² between these vectors as a measure of fit and repeat this measure over 100 iterations, resulting in a distribution of fits. For the between-subject consistency metrics, split halves are computed using the same protocol used for the within-subject consistency. For the between-subject analysis, however, ${\displaystyle {\overrightarrow{v}}_1}$ from subject_is choice behavior is compared to ${\displaystyle {\overrightarrow{v}}_2}$ from subject_js choice behavior (i.e. we generate a random split from each subject to compare, identical to the within-subject protocol). This approach is an alternative to simply computing the fit between two subjects by aggregating over all collected data. We take this random split approach because when all data are used to compare two subjects, this analysis results in a single point estimate of the between-subject consistency—not a distribution of values, as is the case in our protocol. This single-point estimate could overestimate the between-subject correspondence, in relation to the within-subject measure. Instead, estimating a random split for each subject and then comparing each subjects’ data results in a distribution of scores, which provides a measure not only of the average subject-subject correspondence, but also a measure of the variance of the correspondence between subjects (i.e. variation over random splits). Moreover, this approach ensures that both the within- and between-subject correspondence measures are equally powered (i.e. there are not more samples used to compare between subjects, resulting in a biased estimation of between-subject correspondence).”

Reviewer #2 (Recommendations for the authors):

The goal of this paper is to use a model-based approach, developed by one of the authors and colleagues in 2021, to critically re-evaluate the claims made in a prior paper from 2018, written by the other author of this paper (and colleagues), concerning the role of perirhinal cortex in visual perception. The prior paper compared monkeys with and without lesions to the perirhinal cortex and found that their performance was indistinguishable on a difficult perceptual task (categorizing dog-cat morphs as dogs or cats). Because the performance was the same, the conclusion was that the perirhinal cortex is not needed for this task, and probably not needed for perception in general, since this task was chosen specifically to be a task that the perirhinal cortex *might* be important for. Well, the current work argues that in fact the task and stimuli were poorly chosen since the task can be accomplished by a model of the ventral visual cortex. More generally, the authors start with the logic that the perirhinal cortex gets input from the ventral visual processing stream and that if a task can be performed by the ventral visual processing stream alone, then the perirhinal cortex will add no benefit to that task. Hence to determine whether the perirhinal cortex plays a role in perception, one needs a task (and stimulus set) that cannot be done by the ventral visual cortex alone (or cannot be done at the level of monkeys or humans).

There are two important questions the authors then address. First, can their model of the ventral visual cortex perform as well as macaques (with no lesion) on this task? The answer is yes, based on the analysis of this paper. The second question is, are there any tasks that humans or monkeys can perform better than their ventral visual model? If not, then maybe the ventral visual model (and biological ventral visual processing stream) is sufficient for all recognition. The answer here too is yes, there are some tasks humans can perform better than the model. These then would be good tasks to test with a lesion approach to the perirhinal cortex. It is worth noting, though, that none of the analyses showing that humans can outperform the ventral visual model are included in this paper - the papers which showed this are cited but not discussed in detail.

Major strength:

The computational and conceptual frameworks are very valuable. The authors make a compelling case that when patients (or animals) with perirhinal lesions perform equally to those without lesions, the interpretation is ambiguous: it could be that the perirhinal cortex doesn't matter for perception in general, or it could be that it doesn't matter for this stimulus set. They now have a way to distinguish these two possibilities, at least insofar as one trusts their ventral visual model (a standard convolutional neural network). While of course, the model cannot be perfectly accurate, it is nonetheless helpful to have a concrete tool to make a first-pass reasonable guess at how to disambiguate results. Here, the authors offer a potential way forward by trying to identify the kinds of stimuli that will vs won't rely on processing beyond the ventral visual stream. The re-interpretation of the 2018 paper is pretty compelling.

We thank the Reviewer for the careful reading of our manuscript and for providing a fantistics synthesis of the current work.

Major weakness:

It is not clear that an off-the-shelf convolution neural network really is a great model of the ventral visual stream. Among other things, it lacks eccentricity-dependent scaling. It also lacks recurrence (as far as I could tell).

We agree with the Reviewer completely on this point: there is little reason to expect that off-the-shelf convolutional neural networks should predict neural responses from the ventral visual stream, for the reasons outlined above (no eccentricity-dependent scaling, no recurrence) as well as others (weight sharing is biologically implausible, as well as the data distributions and objective functions use to optimize these models). Perhaps surprisingly, these models do provide quantitatively accurate accounts of information processing throughout the VVS; while this is well established within the literature, we were careless to simply assert this as a given without providing an account of these data. We appreciate the Reviewer for making this clear and we have changed the manuscript in several critical ways in order to avoid making unsubstantiated claims in the current version. We hope that these changes also make it easier for the casual reader to appreciate the logic in our analyses.

First, in the introduction, we outline some of the prior experimental work that demonstrates how deep learning models are effective proxies for neural responses throughout the VVS. We also demonstrate this model-neural fit in the current paper using electrophysiological recordings (more on that below), but also including comments about the limitation of these models raised by the Reviewer:

“In recent years, deep learning computational methods have become commonplace in the vision sciences. Remarkably, these models are able to predict neural responses throughout the primate VVS directly from experimental stimuli: given an experimental image as input, these models (e.g. convolutional neural networks, CNNs) are able to predict neural responses. These ‘stimulus-comptable’ methods currently provide the most quantitatively accurate predictions of neural responses throughout the primate VVS (Bashivan et al., 2019; Khaligh-Razavi and Kriegeskorte, 2014; Rajalingham et al., 2018; Yamins et al., 2014). For example, early model layers within a CNN better predict earlier stages of processing within the VVS (e.g. V4; Fig. 1b: left, grey) while later model layers better predict later stages of processing 50 within the VVS (e.g. IT; Fig. 1b: left, green). We note that there is not a 1-1 correspondence between these models and the primate VVS as they typically lack known biological properties (Doerig et al., 2022; Zhuang et al., 2021). Nonetheless, these models can be modified to evaluate domain-specific hypotheses (Doerig et al., 2022)—e.g. by adding recurrence (Kietzmann et al., 2019; Kubilius et al., 2018) or eccentricity-dependent scaling (Deza and Konkle, 2020; Jonnalagadda et al., 2021).”

In the introduction we also more clearly demarcate prior contributions from our recent computational work, and highlight how models approximate the performance supported by a linear readout of the VVS, but fail to reach human-level performance:

“Recently, Bonnen et al., 2021 leveraged these ‘VVS-like’ models to evaluate the performance of PRC- intact/-lesioned human participants in visual discrimination tasks. While VVS-like models are able to approximate performance supported by a linear readout of high-level visual cortex (Fig. 1b: right, green), human participants are able to out outperform both VVS-like models and a linear readout of direct electrophysiological recordings from the VVS (Fig. 1b: right, purple). Critically, VVS-like models approximate PRC-lesioned performance. While these data implicate PRC in visual object processing,…”

Results from these analyses were essential to understanding the logic of the paper but previously (as noted by the Reviewer) this critical evidence was cited but not directly presented. We include a description to these we describe these data in the introduction more thoroughly, and substantially change figure one, in order to visualize these data (b)

Moreover, we include a over of the methods and data used to generate these plots in the results and methods sections (only showing the results (lines 85-112), for brevity):

While there is little reason to expect that off-the-shelf convolutional neural networks should predict neural responses from the ventral visual stream, we believe that these modifications to the manuscript (to the introduction and figure one, as well as the results and methods sections) make clear that these models are nonetheless useful methods for predicting VVS responses and the behaviors that depend on the VVS.

To the authors' credit, they show detailed analysis on an image-by-image basis showing that in fine detail the model is not a good approximation of monkey choice behavior. This imposes limits on how much trust one should put in model performance as a predictor of whether the ventral visual cortex is sufficient to do a task or not. For example, suppose the authors had found that their model did more poorly than the monkeys (lesioned or not lesioned). According to their own logic, they would have, it seems, been led to the interpretation that some area outside of the ventral visual cortex (but not the perirhinal cortex) contributes to perception, when in fact it could have simply been that their model missed important aspects of ventral visual processing. That didn't happen in this paper, but it is a possible limitation of the method if one wanted to generalize it. There is work suggesting that recurrence in neural networks is essential for capturing the pattern of human behavior on some difficult perceptual judgments (e.g., Kietzmann et al 2019, PNAS). In other words, if the ventral model does not match human (or macaque) performance on some recognition task, it does not imply that an area outside the ventral stream is needed - it could just be that a better ventral model (eg with recurrence, or some other property not included in the model) is needed. This weakness pertains to the generalizability of the approach, not to the specific claims made in this paper, which appear sound.

We could not agree more with the Reviewer on these points. It could have been the case that these models' lack of correspondence with known biological properties (e.g. recurrence) led them to lack something important about VVS-supported performance, and that this would derail the entire modeling effort here. Surprisingly, this has not been the case, as is evident in the clear correspondence between model performance and monkey data in Eldridge et al. 2018. Nonetheless, we would expect that other experimental paradigms should be able to reveal these model failings. And future work evaluating PRC involvement in perception must contend with this very problem in order to move forward with this modeling framework. That is, it is of critical importance that these VVS models and the VVS itself exhibit similar failure modes, otherwise it is not possible to use these models to isolate behaviors that may depend on PRC.

A second issue is that the title of the paper, "Inconsistencies between human and macaque lesion data can be resolved with a stimulus-computable model of the ventral visual stream" does not seem to be supported by the paper. The paper challenges a conclusion about macaque lesion data. What inconsistency is reconciled, and how?

It appears that this point was lost in the original manuscript; we have tried to clarify this idea in both the abstract and the introduction. In summary, the cumulative evidence from the human lesion data suggest that PRC is involved in visual object perception, while there are still studies in the monkey literature that suggest otherwise (e.g. Eldridge et al. 2018). In this manuscript, we suggest that this apparent inconsistency is, in fact, simply a consequence of reliance on information interpretations of the monkey lesion data.

We have made substantive changes to the abstract so this is an obvious, central claim.

We have also made substantive changes to the introduction to make resolving this cross-species discrepancy a more central aim of the current manuscript, (lines 56-83)

Author details

1. Tyler Bonnen

   Stanford University, Stanford, United States

   Contribution

   Conceptualization, Formal analysis, Visualization, Methodology, Writing – original draft, Writing – review and editing

   For correspondence

   bonnen@stanford.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-8709-1651

2. Mark AG Eldridge

   Laboratory of Neuropsychology, National Institute of Mental Health,National Institutes of Health, Bethesda, United States

   Contribution

   Resources, Data curation, Supervision, Methodology, Writing – original draft, Writing – review and editing

   For correspondence

   mark.eldridge@nih.gov

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-4292-6832

• 521

  Page views

• 47

  Downloads

• 0

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.