An overview of the user interface and functionalities.

a) The interface for annotation of datasets. In this example, a tomographic slice has been annotated with various features – a detailed explanation follows in Fig. 5. b) After annotation, multiple neural networks (or ‘models’) are set up and trained on the aforemention ed annotations. The models can then be used to segment the various distinct features. In this example, double membrane vesicles (DMVs), single membrane vesicles, ribosomes, intermediate filaments, mitochondrial granules, and molecular pores in the DMVs are segmented. c) After training or downloading the required models and exporting segmented volumes, the resulting segmentation are immediately available within the software for 3d rendering and inspection. d) The Ais model repository at aiscryoet.org facilitates sharing and reuse of trained models. After validation, submitted models can be freely downloaded by anyone. e) Additional information, such as the pixel size and the filtering applied to the training data, is displayed alongside all models in the repository, in order to help a user identify whether a model is suited to segment their datasets.

Comparison of some of the default models available in Ais.

a The computational cost is only roughly proportional to the number of model parameters, which is reported in the software. The specifics of the model architecture affect the processing speed more significantly. b Time required to process one 511×720 pixel sized tomographic slice. c The loss, calculated as the binary cross-entropy (bce) of predicted and original annotation, is a (rough) metric of how well a model performs (see Methods). From left to right, the columns list the losses of the membrane, carbon, and antibody platform models, respectively. d Unlike the other models, Pix2 pix is not trained to minimize the bce loss but uses a different loss function instead. The bce loss values shown here were computed after training and may not be entirely comparable.

A comparison of different neural networks for tomogram segmentation.

a) A representative example of the manual segmentation used to prepare training datasets. Membranes are annotated in cyan, carbon film in yellow, and antibody platforms in magenta. For the antibody training set, we used annotations prepared in multiple slices of the same tomogram, but for the carbon and membrane training set the slice shown here comprised all the training data. b) A tomographic slice from a different tomogram that contains the same features of interest, also showing membrane bound antibodies with elevated Fc platforms that are adjacent to carbon (magenta arrowheads). c) Results of segmentation of membranes (top; cyan), carbon (middle; yellow), and antibody platforms (bottom; magenta), with the six different neural networks.

Model interactions can significantly increase segmentation accuracy.

a) An overview of the settings available in the ‘Models’ menu in Ais. Three models: 1) ‘membrane’ (red), 2) ‘carbon’ (white), and 3) ‘antibody platforms’ (green) are active, with each showing a different section of the model settings: the training menu (1), prediction parameters (2), and the interactions menu (3). b) A section of a tomographic slice is segmented by two models, carbon (white; parent model) and membrane (red; child model), with the membrane model showing a clear false positive prediction on an edge of the carbon film (panel ‘without interactions’). By configuring an avoidance interaction between the membrane model that is conditional upon the carbon model’ s prediction, this false positive is avoided (panel ‘with interactions’). c) By setting up multiple model interactions, inaccurate predictions by the ‘antibody platforms’ model are suppressed. In this example, the membrane model avoids carbon while the antibody model is set to colocalize with the membrane model. d) 3D renders (see Methods) of the same dataset as used in Fig. 2 processed three ways: without any interactions (left), using model competition only (middle), or by using model competition as well as multiple model interactions (right).

Automated particle picking for sub-tomogram averaging of antibody complexes.

a) Manually prepared annotations used to train a neural network to recognize antibody platforms (top) or antibody-C1 complexes (bottom). b) Segmentation results as visualized within the software. Membranes (red) and carbon support film (white) were used to condition the antibody (green) and antibody-C1 complex (yellow) predictions using model interactions. c) 3 D representations of the segmented volumes rendered in Ais. d) Tomographic slices showing particles picked automatically based on the segmented volume shown in panel c. e) Subtomogram averaging result of the 2499 automatically picked antibody platforms. f) Subtomogram averaging result obtained with the 602 automatically picked antibody-C1 complexes. The quadrants in panels e and f show orthogonal slices of the reconstructed density maps and a 3D isosurface model (the latter rendered in ChimeraX).

Segmentations of complex in situ tomograms.

a) A segmentation of seven distinct features observed in the base of C. reinhardtii cilia12 (EMPIAR-11078, tomogram 12): membranes (gray), ribosomes (magenta), microtubule doublets (green) and axial microtubules (green), non-microtubular fi laments within the cilium (blue), interflagellar transport trains (yellow), and glycocalyx (orange). Inset: a perpendicular view of the axis of the cilium. The arrows in the adjacent panel indicate these structures in a tomographic slice. b) A segmentation of six features observed in and around mitochondria in a mouse neuron with Huntington disease phenotype 14 (EMD-29207): membranes (gray), mitochondrial granules (yellow), membranes of the mitochondrial cristae (red), microtubules (green), actin (turquoise), and ribosomes (magenta). c) Left: a segmentation of ten different cellular components found in tomograms of coronavirus infected mammalian cells 16 : double membrane vesicles (DMVs, light red), single membranes (gray), viral nucleocapsid proteins (red), viral pores in the DMVs (blue), nucleic acids in the DMVs (pink), microtubules (green), actin (cyan), intermediate fi laments (orange), ribosomes (magenta), and mitochondrial granules (yellow). Right: a representative slice, with examples of each of the features (except the mitochondrial granules) indicated by arrows of the corresponding colour.