Overview of datasets currently integrated in openretina.

Dataset names are with respect to our huggingface repository. Abbreviations: WN = white noise, MEA = multielectrode array. The reported number of cells is after each dataset’s own quality checks, but prior to possible exclusions based on the Fraction of Explainable Variance (FEV).

openretina - a Python package for ANN-based computational models of the vertebrate retina.

(a) openretina supports modelling of a variety of experimental data across different model organisms, (b) recording modalities (e.g. electrophysiological or imaging recordings), and (c) stimuli (e.g. colour and greyscale, images and movies). (d) openretina allows researchers to use existing models or train their own ((i)); to use models for in silico testing of hypotheses about retinal computations ((ii)), and guiding the design of new experiments ((iii)). (e) The package is designed to be: easy to use, offering an accessible entry point into retina modelling; flexible, using modern deep learning tools; modular, featuring model architectures that allow combining different components (Core + Readout); and expandable, allowing researchers to contribute new datasets and models, and increasing the value of openretina as a resource for the retina research community.

Overview of modelling framework and training pipeline in openretina.

(a) Typical retina experiments involve recordings of neural responses across multiple experimental sessions. Each session includes paired stimulus and response data, such as videos presented to the retina and the corresponding neural activity recorded via multielectrode arrays (MEAs) or two-photon calcium imaging. Predictive models are then trained using the same stimulus-response pairs to reproduce retinal output. (b) Model training workflow. Input data to the model (in orange) for different sessions is processed, split into different folds for training and evaluation, and composed into dataloaders by functions in openretina.data_io. The core modelling stage (in blue) passes these inputs into a full model ( openretina.models ) composed of layers, regularisers, and loss functions ( openretina.modules ), with training managed by PyTorch and Lightning (in pink). After training, evaluation utilities ( openretina.eval, in purple) provide metrics, perform comparisons to ground truth, and support testing on novel stimuli. (c) Mock training code examples for the different stages. For more detailed explanations and working examples, refer to the package repository and online documentation (open-retina.org/package_docs).

Within-dataset model comparison.

Comparison of different model architectures trained on the euler_lab/hoefling_2024 dataset (Höfling et al., 2024) evaluated on the test data. We report Pearson Correlation and Fraction of Explainable Variance Explained (FEVE) across neurons. Correlation is reported for both comparing the model prediction with the average response across test trials (Avg) and to individual trials (Indiv) with ± specifying the standard deviation of the correlation across the individual trials. All models also share the same “time lag”, i.e., the same number of frames consumed by the convolutional kernel along the temporal dimension. All evaluated model assets and hyperparameters are available on huggingface.

openretina provides tools for visualising model weights and for performing in-silico experiments.

(a) Optimised stimuli: The trained model allows us to optimise stimuli towards any objective, e.g. towards activating a particular neuron strongly. The panel shows the spatial and temporal components of an optimised MEI stimulus for a neuron. For the decomposition, we followed Höfling et al. (2024) and ran singular value decomposition on each colour channel of the MEI stimulus individually. The visualisation then shows the first singular vector for the decomposed spatial and temporal dimensions, respectively. (b) Discriminatory stimuli: Discriminatory objective functions are used to optimise a stimulus towards activating certain neurons strongly while inhibiting other groups of neurons. The upper part of the panel shows both an MEI, optimised to activate all transient Suppressed-by-Contrast RGCs (tSbC) cells, and a most discriminatory stimulus (MDS), which was optimised to drive the activity of tSbC cells, while keeping the activity of ON and OFF cells low. As seen by the average responses in the optimisation window (lower sub-plot), while for the MEI stimulus the average response of ON cells is higher than the response of group 28 cells, the MDS leads to a higher response of group 28 cells compared to other ON cells. (c) Internal weights: By zooming in and visualising individual model weights, we can better understand how the neural network reproduces the behaviour of the retina. The upper part of the panel shows a visualisation of the spatial and temporal weights of a three-dimensional convolutional layer. The lower part of the panel shows the readout mask, which determines which spatial location in the stimulus influences a modelled neuron, and the readout channel weights that determine the influence of each convolutional channel on the modelled neuron. (d) Response tuning maps: We can evaluate the model neurons along directions in stimulus space that we are interested in by changing stimulus parameters corresponding to those directions and examining model responses. In this example, we examined the directions along which green and UV temporal contrast selectivities, respectively, change from negative to positive. Lines represent iso-response curves (Gollisch and Herz, 2012), arrows are response gradient directions.

Example application of in silico experiments.

(a) MEIs: Green channel participation to the spatial component of the initialisation of MEI optimisation algorithm and associated identified MEI after convergence for an example cell. (b) Local gradients: Example of image/gradient pairs for two images. This data-driven estimation of the gradient captures the local structure of the cell’s sensitivity in different stimulus context. (c) Gradient field: Gradient field (green channel) of the same cell. Images in A and B are projected on the vector field and represented using coloured symbols. Arrows sketch potential directions the optimising algorithm could have taken to reach predicted MEIs. Background colour indicates the relative firing rate elicited by stimuli falling in this part of the stimulus space. Insets at the extremes of the axes correspond to the principal components (PCs) (and their negative) of the PCA analysis across all gradients of an example cell.

Cross-dataset model comparison.

Evaluation results of pre-trained models provided via huggingface. All models use a base, non-recurrent convolutional architecture, and we only include models that were trained on natural scenes or natural movies datasets. To enhance comparability, on all datasets we filter out cells with a FEV lower than 0.15 and report the number of test stimuli repeats, along with different dataset metadata. All models trained on natural movies also share the same “time lag”, i.e., the same number of frames consumed by the convolutional kernel along the temporal dimension. Datasets indicated by “*” use static natural images rather than movies, and the corresponding models operate on single frames without a temporal dimension. All evaluated model assets and hyperparameters are available on huggingface.

FEVE discriminates between models with the same correlation performance

(a) (Top) Predicted responses of Model A (orange) and Model B (blue) for an example neuron from the Höfling et al. (2024) dataset on a portion of the test set, overlaid on the trial-averaged response (black) and inter-trial variability (shaded, ±1 s.d.). Model B is the high resolution CNN in Table 2, and Model A the low resolution CNN. ΔFEVE and ΔCorr denote the difference in fraction of explainable variance explained and Pearson correlation (Model B − Model A), respectively. (Bottom) Temporal decomposition of the FEVE difference between models. The filled area (right axis) shows the pointwise contribution at each time point, defined as , where is the trial-averaged response, is the explainable variance, and T is the number of time points. This normalization ensures the sum over all time points equals Δ FEVE, since where is the temporal mean of the squared errors. Positive values (blue) indicate time points where Model B has lower squared error; negative values (orange) indicate where Model A is better. The black line (left axis) shows the running cumulative sum, which converges to the total ΔFEVE. This decomposition follows from the fact that the noise variance terms cancel when differencing two FEVE values computed on the same neuron (Cadena et al., 2019). (b) Same as (a), but on models trained on the Sridhar et al. (2024) dataset. Model A and B share the same architecture. B is after hyperparameter tuning.