Tracking prey capture behavior in five species of fish larvae. (A) Phylogenetic relationship between percomorph species used in this study, and their relationship to zebrafish (a cyprinid). Based on Betancur-R et al. (2017). (B) Schematics of larvae of each species studied here (and zebrafish, Danio rerio, for reference). Scale bar: 1 mm. (C) Outline of tracking procedure. Raw video frames were analyzed with three different neural networks to extract tail pose, eye pose and prey position in each frame. Kinematic features were extracted from raw pose data and used for subsequent analyses. (D-F) Tracking output for each neural network for a single frame from a recording of an L. attenuatus larva, showing identified artemia (D), tail points (E) and eye points (F). (G) One minute of tail and eye tracking from L. attenuatus showing tail tip angles (top trace) and right and left eye angles (bottom traces) relative to midline. Bottom trace, gray shaded boxes: automatically identified prey capture periods, when the eyes are converged (see Figure 2). Top trace, shaded boxes: automatically identified and classified bouts. Color corresponds to cluster identity in Figure 4. (H) Cumulative distribution of bout lengths across species for all fish combined. Colors as in (A) and (B). Medaka bouts (purple line) are longer than cichlid bouts.

Eye movements during prey capture and statistics of hunting sequences. (A) Histograms of eye convergence angles for each species. The convergence angle is the angle between the long axes of the eyes. Shaded bars: normalized binned counts of convergence angles from all fish. Black lines: a best fit gaussian mixture model for each species (with one or two components). For all cichlid species, the data are better modeled as being drawn from two underlying distributions. For medaka (O. latipes), the data are better modeled with a single underlying distribution. (B) Improvement in fit for a two component over single component Gaussian mixture model, assessed using Bayesian inference criterion (BIC). The BIC is a measure of model fit, while punishing over-fitting. Lower values are better. Two mixtures provide a better fit over a single mixture for all species except medaka (OL). (C) Proportion of time spent by cichlids engaged in prey capture within the first five minutes of being introduced to the behavior arena. Points are single animals; black bar is the median. (D) Hunting rate, measured as the number of times eye convergence is initiated per minute, within the first five minutes. Points are single animals; black bar is the median. (E) Kernel density estimation of hunt durations for each species (all animals pooled). A. burtoni (pink) hunts skew shorter, L. attenuatus (brown) hunts tend to be longer. (F) Median hunt duration for each animal compared across species. Black bar is the median across animals. (G) Capture rate (number of artemia consumed) per minute over the first five minutes for three cichlid species. Black bar is the median across animals.

Location of prey in the visual field during prey capture in cichlids. (A) Trajectories of prey in the visual field for all automatically identified and tracked hunting events. Each prey is represented by a single line that changes in color from blue to yellow from the onset of eye convergence to when the eyes de-converge. Scale bar: 5 mm. (B) Kernel density estimation of the distribution of prey in the visual field across all hunting events. Rows: individual species. Columns: snapshots showing the distribution of hunted prey items at the beginning, middle and end of hunting sequences. Scale bar: 2 mm. (C-D) Kernel density estimation of prey distance (C) and azimuthal angle from the midline (D) at the onset (top), in the middle (center) and at the end (bottom) of hunting episodes. Each column shows the distribution from all events for a single species.

Interspecies comparison of prey capture strategies. (A) Cumulative explained variance for the “canonical” principal components (PCs) obtained from all species (black dotted line), and PCs for each species individually (colored lines). In all cases, three PCs explain >90% of the variance in tail shape. (B) “Eigenfish” of the first three canonical PCs. Each principal component represents a vector of angles from the base to the tip of the tail (oriented with the fish facing up). At a given moment, the shape of the tail can be described as a linear combination of these vectors. Colors correspond to the tail shape obtained by scaling each PC from -4 to 4 standard deviations from the mean. (C) Each species’ eigendecomposition compared against the canonical PCs computed for all species together. Color intensity represents the cosine similarity between pairs of vectors. The strong diagonal structure (particularly in the first three PCs) shows that similar PCs are obtained by analyzing species separately or together. (D) Behavioral space of L. attenuatus. Each point represents a single bout. Bouts are projected onto the first three PCs, aligned to the peak distance from the origin in PC space and then projected into a two-dimensional space using TSNE. Color intensity represents density of surrounding points in the embedding. (E) Clustered behavioral space of L. attenuatus. Clusters (colors) are computed via affinity propagation independently of the embedding. (F) Prey capture and spontaneous bouts in L. attenuatus. Prey capture score is the probability that the eyes are converged at the peak each bout. Bouts are colored according to the mean prey capture score for their cluster. Blue: clusters of bouts that only occur during spontaneous swimming; red: clusters of bouts that only occur during prey capture. (G-H) Example prey capture clusters from L. attenuatus (G) and medaka, O. latipes (H). For each cluster, top left: mean rostrocaudal bending of the tail over time; bottom left: time series of tail pose projected onto first three PCs (mean +/- standard deviation); bottom right: reconstructed tail shape over time for mean bout. (I) Representative frames of an L. attenuatus capture spring (left) and medaka side swing (right), highlighting differences in tail curvature between these behaviors. (J) Location of prey in the visual field (black dots) immediately prior to the onset of a side swing. All events mirrored to be on the right. X marks the midpoint of the eyes, aligned across trials. (K) Confusion matrix of clustered hunt termination bouts from all species. Termination bouts from all species were sorted into five clusters based on their similarity. Rows show the proportion of bouts from each species that were assigned to each cluster (columns). Cichlid bouts are mixed among multiple clusters, while medaka bouts (OL) mostly sorted into a single cluster. (L) Capture strikes in cichlids. Representative examples of tail kinematics during attack swims (left) and capture springs (right) from each species, including time series of tail pose projected onto the first three PCs (mean +/- standard deviation, all species combined) shown for each type of strike. (M) Two hypotheses for distance estimation make different predictions of how heading (black dotted line) changes over time as fish approach prey (pink star). Top: fish maintain prey in the central visual field and use binocular cues to judge distance. Bottom: fish “spiral” in towards prey, using motion parallax to determine distance. Black arrows indicate motion of prey stimulus across the retina. (N) Change in heading over time leading up to a capture strike for L. attenuatus (brown) and medaka (purple). Left: time series of heading. Zero degrees represents the heading 25 ms prior to the peak of the strike (black dotted line). Mean ± s.e.m. across hunting events. Right: comparison of the rate of heading change, computed as the slope of a line fit to each hunting event 200-25 ms prior to the peak of the strike. The heading decreases more rapidly leading up to a strike in medaka than in L. attenuatus.

Schematic of hunting strategies in teleost larvae. Top: prey capture in zebrafish larvae (Danio rerio) begins with eye convergence and a J-turn to orient towards the prey. The prey is then approached with a series of low amplitude approach swims. Once in range in the central visual field, the prey is captured with an attack swim or an S-strike. Middle: prey capture in cichlids (represented by L. attenuatus) also begins with eye convergence and a J-turn. Prey is approached with a wide variety of tail movements. The prey is captured when it is in the central visual field with either an attack swim, or a capture spring, during which the tail coils over several hundreds of milliseconds. Bottom: prey capture in medaka begins with reorienting J-like turns, but these do not centralize prey in the visual field. Instead, the prey is kept lateral in the visual field and is captured with a side swing.