Workflow diagram of CHROMAS.

Dotted lines indicate optional steps in the process, allowing flexibility depending on experimental requirements. Orange dots represent new functions that did not exist in the initial versions of the pipeline (Reiter et al., 2018; Woo, Liang et al., 2023). Each step of the pipeline can be run independently, providing users with the flexibility to execute specific stages as needed. Additionally, CHROMAS includes options to generate visual output videos, enabling users to assess and validate the accuracy and quality of the result at every step.

Chunking process.

The video frames are sorted based on their focus score. Images that are too dark or too blurry are removed from the dataset, and chunks made of consecutive sharp frames are created. In the figure above, the score decreases due to a brightness drop at frame 1200 and to the subject going out of focus around frame 3500, resulting in three distinct chunks. Continuity between chunks is reestablished during the “Stitching” process (see Fig. 5).

Segmentation results.

In each frame, each pixel is classified as either part of a chromatophore or not. a1-2) Original frame before segmentation. b1-2) Original frames with edges of segmented chromatophores in green. c1-2) Binary segmentation with chromatophores in black and background in white. Note that even the smallest chromatophores, though faint and difficult to discern on the original footage, are segmented.

Registration.

a) Overlay of two frames 20 frames apart within the same chunk. Panels (a1-a2) illustrate the misalignment of two chromatophores (red and blue) before registration. Panels (b1-b2) show the same frames after automated registration. The registration process compensates for small movements or drifts over time, ensuring correct identification and tracking of the chromatophores over time.

Stitching video chunks.

This figure illustrates the stepwise refinement process for stitching adjacent video chunks, ensuring precise alignment critical for downstream analyses. a) Masterframe of chunk 1, with an initial ellipse fit shown in red. b) Masterframe of chunk 2, featuring the ellipse fit and a grid of points for fine alignment in blue. c) Chunk 2 aligned on chunk 1 using the ellipse-fitting method. d) First round of fine alignment, maximizing phase correlation between patches surrounding the points. Points prior to fine alignment are shown in blue, and their adjusted positions post-alignment are in pink. e) Second round of fine alignment, from pink to red. f) Final overlay between chunk 1 and the fully aligned chunk 2. In black are the chromatophore areas common to both chunks; in red and blue are chromatophore areas that are unique to chunk 1 and chunk 2, respectively.

Calculation of chromatophores areas.

a) Masterframe and zoom on the segmentation of one chromatophore’s contractions and expansions. b) Cleanqueen. Each color represents a different chromatophore territory. c) Surface area of each segmented chromatophore over time (colors as in b). The chromatophore in a is highlighted in the thick blue line. This approach measures the total area occupied by a chromatophore within its defined territory, and offers no information about its shape.

a) A single chromatophore (the same one as in fig. 6), is divided radially into 36 slices. b) Close-up on one of the 36 slices, indicating how its surface area is estimated. c) Chromatophore “epicenters” (see text) in blue and motion markers (see text) in green, with edges for triangulation. d) Plot representing the surface areas of the 36 slices of this chromatophore over time. Note that the dynamics of a chromatophore are now described by 36 values per frame, rather than one, as in Fig 6c. Note also that the top half of the traces describing this chromatophore deviate clearly from the others around frames 550 and 960, indicating their differential control. This fine-grain description of chromatophore deformations over time is used to reveal the fine details of their individual and collective motor control.

Tracking of chromatophores over 12 days of development (t₀ = 7 days post hatching (dph) and t₁ = 21 dph).

During this time, the number of chromatophores nearly doubled, with many changing chromatic properties. Despite these changes and intermittent filming, our pipeline reliably tracks individual chromatophores, as illustrated with two arbitrarily selected chromatophores, in blue and green.

Example of 4K footage of Sepia officinalis processed through the CHROMAS pipeline.

a1-2) Original video frame. b1-2) Binary segmentation with chromatophores in black and background in white. c1-2) Epicenters of chromatophores in blue and motion markers with triangulation in green. d1-2) Putative motor units calculated using the Affinity Propagation algorithm. In this example, PCA-ICA analysis revealed independent components influencing the activity of each chromatophore. We found 4.68 ± 1.41 (mean ± std) ICs per chromatophore (n=1616). Those independent components were then clustered together based on their covariation and plotted with the same color. For visualisation purposes, only clusters of size 8 and 9 are plotted. In this segment, we found 10.37 ± 14.23 (mean ± std) chromatophores per cluster, with 90% of clusters consisting of less than 20 chromatophores. This example highlights CHROMAS’s ability to process complex datasets, such as footage of Sepia officinalis, which exhibits a high density of chromatophores and very active dynamics correlated with its camouflage and other behaviors.

Performance on a workstation with a 10 core Intel Core i9-10900X with 64 GB RAM and a NVIDIA RTX A4000 GPU.

Runtime in milliseconds per frame (mean ± s.d.).

Performance on a laptop with a 4 core Intel Core i5-1135G7 with 16 GB RAM and no NVIDIA GPU.

Runtime in milliseconds per frame (mean ± s.d.).