Generation of in vitro and in vivo live-cell imaging data.

A. Micrographs depicting mammary epithelial MCF10A cells transduced with H2B-miRFP703 marker and grown to form a confluent monolayer. The monolayer was acquired with a fluorescence microscope for several hours with 1-, 2- or 5-min time resolution. B. The centroid (x, y) and the time (t) of apoptotic events were annotated manually based on morphological features associated with apoptosis. Non-apoptotic cells were identified by automatic segmentation of nuclei. C. Image time-lapses showing a prototypical apoptotic event (upper panels), with nuclear shrinkage and chromatin condensation, and a non-apoptotic event (bottom panels). D. Charts showing the quantification of nuclear size (left) and the standard deviation (SD) of the nuclear pixel intensity (right) of apoptotic and non-apoptotic cells (n = 50). Central darker lines represent the mean, gray shades bordered by light colored lines represent the standard deviation. Nuclear area over time expressed as the ratio between areas at Tn and T0. E. Simplified drawing showing the surgical set-up for lymph node and spleen. F-G. Organs are subsequently imaged with intravital 2-photon microscopy (IV-2PM, F), generating 3D time-lapses (G). H. Representative IV-2PM micrograph and I. selected crops showing GFP-expressing neutrophils (white) undergoing apoptosis. The apoptosis sequence is depicted by raw intensity signal (upper panels) and 3D surface reconstruction (bottom panels).

ADeS, a pipeline for apoptosis detection.

A. ADeS input consists of single channel 2D microscopy videos (x,y,t) B. Each video frame is pre-processed to compute the candidate Regions of Interest (ROI) with a selective search algorithm. C. Given the coordinates of the ROI at time t, ADeS extracts a series of snapshots ranging from t-n to t+n. A deep learning network classifies the sequence either as non-apoptotic (0) or apoptotic (1). D. The predicted apoptotic events are labelled at each frame by a set of bounding boxes which, E. are successively linked in time with a tracking algorithm based on euclidean distance. F. The readout of ADeS consist of bounding boxes and associated probabilities, which can generate a probability map of apoptotic events over the course of the video (left) as well as providing the number of apoptotic events over time (right).

Conv-Transformer architecture at the core of ADeS.

Abstracted representation of the proposed Conv-Transformer classifier. The input sequence of frames is processed with warped convolutional layers, which extract the features of the images. The extracted features are passed into the 4 transformer modules, composed of attention and feedforward blocks. Finally, a multi layer perceptron enables classification between apoptotic and non-apoptotic sequences.

Comparison of deep learning architectures for apoptosis classification.

Comparative table reporting accuracy, F1 and AUC metrics for a CNN, 3DCNN, Conv-LSTM, and Conv-Transformer. The classification accuracy is reported for static frames or image-sequences. N.A. stands for non applicable. The last column shows which cell death study employed the same baseline architecture displayed in the table..

Training and performance in vitro.

A. Confusion matrix of the trained model at a decision-making threshold of 0.5. B. Receiver operating characteristic displaying the false positive rate (specificity) corresponding to each true positive rate (sensitivity). C. Training accuracy of the final model after 100 epochs of training. D. Representative example of apoptosis detection in a time-lapse acquired in vitro. E. Multiple detection of nuclei undergoing apoptosis displays high sensitivity in densely packed field of views. F. Heatmap representation depicting all apoptotic events in a movie and the respective probabilities. G. Bar plots showing the true positive rate (TPR) and false positive rate (FPR) of ADeS applied to five testing movies, each one depicting an average of 98 apoptosis. H. Time course showing the cumulative sum of ground truth apoptosis (blue) and correct predictions (red). I. 2D visualization of spatial-temporal coordinates of ground truth (blue) and predicted apoptosis (red). In the 2D representation, the radius of the circles maps the temporal coordinates of the event. J. Pixel distance between ADeS predictions and the nearest neighbor (NN) of the ground truth (left) in comparison with the NN distance obtained from a random distribution (right). The plot depicts all predictions of ADeS, including true positives and false positives. K. Scatterplot of the spatial distance between ground truth and true positives of ADeS. Ground truth points are centered on the X = 0 and Y = 0 coordinates. L. Distribution of the temporal distance (frames) of the correct predictions from the respective ground truth nearest neighbor. Statistical comparison was performed with Mann-Whitney test. Columns and error bars represent the mean and standard deviation respectively. Statistical significance is expressed as: p ≤ 0.05 (*), p ≤ 0.01 (**), p ≤ 0.001 (***), p ≤ 0.0001 (****).

3D rotation of the in vivo dataset.

A Depiction of a 3D volume cropped around an apoptotic cell. Each collected apoptotic sequence underwent multiple 3D rotation in randomly sampled directions. The rotated 3D images were successively flattened in 2D. B. Gallery showing the result of multiple volume rotations applied to the same apoptotic sequence. The vertical axis depicts the sequence over time, whereas the horizontal describes the rotational degree applied to the volumes.

Training and performance in vivo.

A. Confusion matrix of the trained model at a decision-making threshold of 0.5. B. Receiver operating characteristic displaying the false positive rate (FPR) corresponding to each true positive rate (TPR). C. Training accuracy of the final model trained for 200 epochs with data augmentations. D. Image gallery showing ADeS classification to sequences with different disruption timing. The generated heatmap reaches peak activation (red) at the instant of cell disruption D. Representative snapshots of a neutrophil undergoing apoptosis. Green bounding boxes represents ADeS detection at the moment of cell disruption E. Representative micrograph depicting the detection of two eosinophil undergoing cell death in the spleen (left) and the respective probability heatmap (right). F. ADeS performances expressed by means of true-positive rate (TPR) and false-positive rate (FPR) over a panel of 23 videos. G. TRA measure distribution of the trajectories predicted by ADeS with respect to the annotated ground truth (n = 8) H. Comparison between human and ADeS by means of TPR and FPR on a panel of 5 randomly sampled videos I. Hierarchical clustering of several video parameters producing two main dendrograms (n = 23). The first dendrogram includes videos with reduced sensitivity and is enriched in several parameters related to cell density and signal intensity. J. Graph showing the effect of cell density on the performances expressed in terms of TPR and FPR (n = 13). K. Comparison of the positive predictive value between videos with large and small signal to noise ratio (left), and videos with large and small shortest cell distance (right). L-M. Selected video parameters are combined into a quality score that weakly correlates with the TPR in overall data (M, n = 23) and strongly correlates with the TPR in selected underperforming data (N, n = 8). Statistical comparison was performed with Mann-Whitney test. Columns and error bars represent the mean and standard deviation respectively. Statistical significance is expressed as: p ≤ 0.05 (*), p ≤ 0.01 (**), p ≤ 0.001 (***), p ≤ 0.0001 (****).

Comparison of cell death identification studies.

Table reporting all studies on cell death classification based on machine learning. For each study, we included the reported classification accuracy, the experimental conditions of the studies, the target input of the classifier, and the capability of performing detection on static frames or microscopy time-lapses. Met conditions are indicated with a green check. Moreover, for each study we reported the architecture of the classifier and the number of apoptotic cells in the training set. N.A. stands for not available and indicates that the information is not reported in the study.

Applications for toxicity assay in vitro.

A. Representative snapshots depicting epithelial cells in vitro at 0 and 24 hours after the addition of PBS and three increasing doses of doxorubicin, a chemotherapeutic drug and apoptotic inducer B. Plot showing the number of apoptotic cells detected by ADeS over time for each experimental condition. C-D. Dose-response curves generated from the drug concentrations and the respective apoptotic counts at 5 h. and 24 h.post-treatment. Vertical dashed lines indicates the EC50 concentration. E. Dose-response curve projected from the fit obtained in (D). The predicted curve allows to estimate the response at higher drug concentrations than the tested ones.

Measurement of tissue dynamics in vivo.

A. Intravital 2-photon micrographs showing ADeS detection of an apoptotic neutrophil (Blue, left) and the subsequent recruitment of neighboring cells (right) in the popliteal LN at 19 h. following influenza vaccination. B. Plot showing the distance of recruited neutrophils with respect to the apoptotic coordinates over time (n = 22) C. Plot showing the instantaneous speed of recruited neutrophils over time (n = 22). The dashed vertical lines indicate the instant in which the apoptotic event occurs. Gray area defines the boundaries of maximum and minimum values. D. Schematic drawing showing the intravital surgical set up of a murine spleen after inducing a local laser ablation. E. Intravital 2-photon micrographs showing the recruitment of GFP-expressing neutrophils (Green) and the formation of a neutrophil cluster (red arrows) at 60 min after photo burning induction. F. Application of ADeS to the generation of a spatiotemporal heatmap indicating the probability of encountering apoptotic events in the region affected by the laser damage. The dashed circle indicates a hot spot of apoptotic events.