Celldetective: an AI-enhanced image analysis tool for unraveling dynamic cell interactions

Reviewer #1 (Public review):

Summary:

In this manuscript, Torro et al. presented CellDetective, an open-source software designed for a user-friendly execution of single-cell segmentation, tracking, and analysis of time-lapse microscopy data. The authors demonstrated the applications of the software by measuring NK cell spreading events acquired with reflection interference contrast microscopy (RICM), as well as detecting target cell death events and their interaction with neighboring NK cells in a multichannel widefield microscopy dataset.

Strengths:

The segmentation (StarDist, Cellpose) and tracking (bTrack) modules implemented were based on existing and published software packages. The authors added the event detection, classification, and analysis modules to enable an end-to-end time-lapse microscopy data processing and analysis pipeline, complete with a graphical user interface (GUI). This minimizes the coding experience required from the user. The documentation that accompanies CellDetective is also adequate.

Weaknesses:

Given that the software was designed to improve user experience, such an approach also limits its scope and functionality and is currently capable of handling very specific types of experiments. Additionally, this reviewer has also encountered many technical difficulties (see documented bugs/crashes below) that have prevented an extensive exploration of all the functionality of CellDetective.

Specifics:

(1) The software can only handle 2D 'widefield' time-lapse imaging datasets. It should be noted that many studies that examine cell-cell interactions in vitro also used confocal microscopy and acquired the time-lapse images in 3D z-stacks to enable the reconstruction of entire cell volumes from multiple optical sections along the z-axis.

Given that almost all of the implemented segmentation (StarDist, Cellpose) and tracking (bTrack) packages already support the handling of 3D datasets, it is unclear why CellDetective was designed to only work with 2D datasets.

As noted above, extending the support for 3D images would allow the scope and utility of this software to be further extended for imaging studies acquired in z-stacks. As an example, the dense clustering of effector cells in Figure 4 had prevented accurate segmentation due to the 2D nature of the experimental dataset. More importantly, support for a 3D dataset could also allow for the tracking of fluorescent protein-based sub-cellular as well as membrane protein localization during cell-cell interactions.

Furthermore, it also widens the potential applicability for analyzing datasets from 3D organoid imaging and perhaps even intravital two-photon microscopy.

(2) The software in its current form only allows the broad demarcation of the cells examined into two populations: targets and effectors. This limits the number of cell populations that can be examined for their interactions. It might be more useful to just allow multiple user-defined populations instead of restricting the populations to target and effector cells only.

(3) Similarly, subsetting of each of the populations could be made more intuitive. Although it is possible to define subsets of cells using the "Custom classification" function under the "Measure" module with user-defined parameters, visualization of multiple groups remains unintuitive and it appears that only one custom classified group can be selected and visualized at any given time in the Signal Annotator under Measurement instead of allowing visualization of multiple (custom defined) groups of cells in different colors. It is also unclear how, if possible at all, to visualize a custom group of cells in the Signal Annotator under the Detect Events module.

Software issues:

(4) When initially tested on v1.3.9, the Segment module could not be initiated (with the error message AttributeError: 'WindowsPath' object has no attribute 'endswith' when attempting to run segmentation).
Update: this has been fixed in v1.3.9.post4 dated February 7th, 2025.

(5) Further testing was then performed by downgrading the software to v1.3.1. While testing the ADCC demo experiment (https://celldetective.readthedocs.io/en/latest/adcc-example.html), the workflow was stuck at attempts to initiate the Detect Events step:

AssertionError: No signal matches with the requirements of the model ['dead_nuclei_channel_mean', 'area']. Please pass the signals manually with the argument selected_signals or add measurements. Abort.

(Update: fixed in the latest v1.3.9.post4 version dated February 7th, 2025)

(6) Random bugs causing the software to crash. Example: switching characteristic to 'status_color' in the Signal Annotator under Measurement caused the software to crash (v1.3.9.post4):

TypeError: ufunc 'isnan' is not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule 'safe'

(7) Overall, when exploring the functionality of the software, there have been multiple instances of software crashes when clicking/switching around to show different parameters, etc.

This reviewer understands the difficulties and time involved in bug fixing and hopes that the experience could have been much smoother and that the software behaves much more stably in order to maximize its useability.

Reviewer #2 (Public review):

Summary:

Immune assays enable the analysis of immune responses in vitro. These assays generate time series image data across several experimental conditions. The imaging parameters such as the imaging modality and the number of channels can vary across experiments. A challenge in the field is the lack of (open source) tools to process and analyze these data. R. Torro, et. al. developed an open source end-to-end pipeline for the analysis of image data from these immune assays. The pipeline is designed with a GUI and is suited for experimental biologists with no coding experience. The authors have incorporated several existing methods and tools for individual tasks such as for segmentation and cell tracking, and incorporated them with custom methods where necessary such as for tracking cell state transitions.

Strengths:

(1) The tool is extremely well-documented and easy to install.

(2) Applicable to a wide variety of imaging modalities and analysis.

(3) There are several different options for each step, such as segmentation using traditional methods or deep learning methods, and all the analysis steps are integrated in one place with a GUI. The no-coding requirement makes this a very powerful tool for biologists and has the potential to enable a wide variety of analyses.

Weakness:

(1) It would be good to provide documentation on how to make the tool applicable for applications and analysis other than for immune profiling since most methods integrated here are applicable well beyond immune profiling. For example, a user might want to use the tool just for the segmentation of their IF microscopy-images.

(2) They applied Celldetective to two immune assays. The authors present the results from these assays and use the results to validate their assay. However, they have not included data that demonstrates results obtained via this pipeline are comparable to results obtained with other pipelines and/or if these results are consistent with what is expected in the literature.

Author response:

In view of the suggestions of the referees, we wish to underline that a user can interact with celldetective at two levels: a non-coder can analyse data and train models without coding, but is necessarily offered pre-determined choices and flexibility. An advanced user however has practically limitless flexibility to extend the fully-open source celldetective, aided by its modularity and detailed manual.

Public Reviews:

Reviewer #1 (Public review):

Summary:

In this manuscript, Torro et al. presented CellDetective, an open-source software designed for a user-friendly execution of single-cell segmentation, tracking, and analysis of time-lapse microscopy data. The authors demonstrated the applications of the software by measuring NK cell spreading events acquired with reflection interference contrast microscopy (RICM), as well as detecting target cell death events and their interaction with neighboring NK cells in a multichannel widefield microscopy dataset.

Strengths:

The segmentation (StarDist, Cellpose) and tracking (bTrack) modules implemented were based on existing and published software packages. The authors added the event detection, classification, and analysis modules to enable an end-to-end time-lapse microscopy data processing and analysis pipeline, complete with a graphical user interface (GUI). This minimizes the coding experience required from the user. The documentation that accompanies CellDetective is also adequate.

Weaknesses:

Given that the software was designed to improve user experience, such an approach also limits its scope and functionality and is currently capable of handling very specific types of experiments. Additionally, this reviewer has also encountered many technical difficulties (see documented bugs/crashes below) that have prevented an extensive exploration of all the functionality of CellDetective.

We apologize for the technical difficulties and bugs; the ones mentioned have been already corrected. New users have also tested the installation and reported it to be bug-free.

We fully agree on the compromise that has to be found between user experience and versatility. We have already tested celldetective in other biological contexts, such as microbiology, but made a choice to showcase it in the article for immunological applications. We invite the reader to consult the software documentation and online examples to learn about more options.

Specifics:

(1) The software can only handle 2D 'widefield' time-lapse imaging datasets. It should be noted that many studies that examine cell-cell interactions in vitro also used confocal microscopy and acquired the time-lapse images in 3D z-stacks to enable the reconstruction of entire cell volumes from multiple optical sections along the z-axis.

Given that almost all of the implemented segmentation (StarDist, Cellpose) and tracking (bTrack) packages already support the handling of 3D datasets, it is unclear why CellDetective was designed to only work with 2D datasets.

As noted above, extending the support for 3D images would allow the scope and utility of this software to be further extended for imaging studies acquired in z-stacks. As an example, the dense clustering of effector cells in Figure 4 had prevented accurate segmentation due to the 2D nature of the experimental dataset. More importantly, support for a 3D dataset could also allow for the tracking of fluorescent protein-based sub-cellular as well as membrane protein localization during cell-cell interactions.

Furthermore, it also widens the potential applicability for analyzing datasets from 3D organoid imaging and perhaps even intravital two-photon microscopy.

We thank the reviewer for this suggestion. Indeed, extension to 3-dimensions is a natural development, since we have chosen segmentation and tracking methods which are compatible with 3D. However, two important strengths of celldetective are: harnessing statistical power of cell populations together with multiplexing biological conditions, and dynamic analysis of fast events.

For both, 2D is advantageous. Our own focus is on analyzing cellular events with minute time resolution, relevant in immunology. By our estimate (experience and literature), 3D timelapse acquisition would reduce the time resolution, as well as throughput (in terms of events and conditions) to below acceptable level. While we don’t envisage this upgrade in the immediate future, we encourage advanced users to contribute to further develop the open-source code in this direction. As a mitigation solution, a 2.5D approach on a flat sample by combining two z planes (in order to address issues of cell superposition for example), could be readily implemented with minimal change.

(2) The software in its current form only allows the broad demarcation of the cells examined into two populations: targets and effectors. This limits the number of cell populations that can be examined for their interactions. It might be more useful to just allow multiple user-defined populations instead of restricting the populations to target and effector cells only.

We thank the reviewer for this suggestion. There is little architectural limitation to its implementation; this will be proposed in the future version. This updated version will allow more than two user-defined populations, labelled directly by the user, which will also facilitate the natural extension to more varied biological applications. Three-way interactions are much more complex, and, to our knowledge, not currently addressed by biologists. The interactions will for the moment be limited to 2 populations interactions, as multipartite ones involve a higher level of code modifications, not immediately envisaged.

(3) Similarly, subsetting of each of the populations could be made more intuitive. Although it is possible to define subsets of cells using the "Custom classification" function under the "Measure" module with user-defined parameters, visualization of multiple groups remains unintuitive and it appears that only one custom classified group can be selected and visualized at any given time in the Signal Annotator under Measurement instead of allowing visualization of multiple (custom defined) groups of cells in different colors. It is also unclear how, if possible at all, to visualize a custom group of cells in the Signal Annotator under the Detect Events module.

The simultaneous visualization of several classes poses problems in the choice of colors and symbols, and may render the tool difficult to use. The time propagation option in the classification tool allows to define event classes as opposed to groups, that are compatible with the Signal Annotator. For more complex classifications, a simple solution is to work with composite classifications, which are already supported by using logical AND/OR operators on the condition defining the class. We believe that this feature is sufficient to address this issue.

Software issues:

(4) When initially tested on v1.3.9, the Segment module could not be initiated (with the error message AttributeError: 'WindowsPath' object has no attribute 'endswith' when attempting to run segmentation).

Update: this has been fixed in v1.3.9.post4 dated February 7th, 2025.

(5) Further testing was then performed by downgrading the software to v1.3.1. While testing the ADCC demo experiment (https://celldetective.readthedocs.io/en/latest/adcc-example.html), the workflow was stuck at attempts to initiate the Detect Events step:

AssertionError: No signal matches with the requirements of the model ['dead_nuclei_channel_mean', 'area']. Please pass the signals manually with the argument selected_signals or add measurements. Abort.

(Update: fixed in the latest v1.3.9.post4 version dated February 7th, 2025)

(6) Random bugs causing the software to crash. Example: switching characteristic to 'status_color' in the Signal Annotator under Measurement caused the software to crash (v1.3.9.post4):

TypeError: ufunc 'isnan' is not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule 'safe'

(7) Overall, when exploring the functionality of the software, there have been multiple instances of software crashes when clicking/switching around to show different parameters, etc.

This reviewer understands the difficulties and time involved in bug fixing and hopes that the experience could have been much smoother and that the software behaves much more stably in order to maximize its useability.

We apologize again for the various technical issues encountered during the review process, and thank the reviewer for mentioning that several bugs were already fixed in the last software release. The open source and software maintenance protocol enabled by github should help to resolve any further emerging issue.

Reviewer #2 (Public review):

Summary:

Immune assays enable the analysis of immune responses in vitro. These assays generate time series image data across several experimental conditions. The imaging parameters such as the imaging modality and the number of channels can vary across experiments. A challenge in the field is the lack of (open source) tools to process and analyze these data. R. Torro, et. al. developed an open source end-to-end pipeline for the analysis of image data from these immune assays. The pipeline is designed with a GUI and is suited for experimental biologists with no coding experience. The authors have incorporated several existing methods and tools for individual tasks such as for segmentation and cell tracking, and incorporated them with custom methods where necessary such as for tracking cell state transitions.

Strengths:

(1) The tool is extremely well-documented and easy to install.

(2) Applicable to a wide variety of imaging modalities and analysis.

(3) There are several different options for each step, such as segmentation using traditional methods or deep learning methods, and all the analysis steps are integrated in one place with a GUI. The no-coding requirement makes this a very powerful tool for biologists and has the potential to enable a wide variety of analyses.

Weakness:

(1) It would be good to provide documentation on how to make the tool applicable for applications and analysis other than for immune profiling since most methods integrated here are applicable well beyond immune profiling. For example, a user might want to use the tool just for the segmentation of their IF microscopy-images.

This is an important suggestion that we will implement as short demonstrations using data from the public domain. These will be proposed as examples in the online documentation.

(2) They applied Celldetective to two immune assays. The authors present the results from these assays and use the results to validate their assay. However, they have not included data that demonstrates results obtained via this pipeline are comparable to results obtained with other pipelines and/or if these results are consistent with what is expected in the literature.

In the final version of the article, we shall compare celldetective with existing literature, including our previous work, when possible. However, we emphasize that most of the presented data are original and don’t have any published equivalent in the literature. Concerning the immunotherapy assays, data presented already show expected trends (see for example Fig. 2 and Fig. 5). We reserve for future publications the systematic comparison with traditional (non microscopy-based) methods, as we consider it out-of-scope here. Additionally, there is, to our knowledge no existing open pipeline performing the full end-to-end analysis.