Analysis of Larval Fish Feeding Behavior under Naturalistic Conditions
Shir Bar
Liraz Levy
Shai Avidan
Roi Holzman
[Paper]
[Poster]
[Data]
[GitHub]
Can you spot the feeding fish?

Abstract

Modern research efforts concerned with animal behavior rely heavily on image and video analysis. While such data are now quick to obtain, extracting and analyzing complex behaviors under naturalistic conditions from digital imagery is still a major challenge. In this study, we introduce a novel system for imaging and analyzing the feeding behavior of freely-swimming fish larvae, as filmed in large aquaculture rearing pools, a naturalistic environment. We first describe the design of a specialized system for imaging these tiny, fast-moving creatures. We then show that an analysis pipeline based on action classification can be used to successfully detect and classify the sparsely-occurring behavior of the fish larvae in a curated experimental setting from videos featuring multiple animals. Additionally, we introduce three new annotated datasets of underwater videography, in a curated and an uncurated setting. Finally, we share the methods used and insights gained during the training and evaluation of models on low sample sizes and highly imbalanced datasets. As these challenges are common to the analysis of animal behavior under naturalistic conditions, we believe our findings can contribute to the growing field of computer vision for the study and understanding of animal behavior.

Overview
An overview of our workflow from data acquisition, through data annotation & curation to data analysis.

We are interested in detecting larval fish feeding behavior in the wild. This behavior is crucial to their survival and has never before been directly imaged outside the laboratory. In order to do this we tackle 3 challenges:
  1. Data acquisition - how do we image tiny, fast moving animals? Details on the data acquisition system can be found in our paper.
  2. Data analysis - how do we automate the detection of feeding behavior? Our data analysis pipeline is presented below and code is available in our Github repo.
  3. Rare events - larval fish eat at a naturally low rate, how does this affect us? Rare events naturally lead to a small sample size and high data imbalance, we use a simulation study to show how these effect model evaluation metrics. More details in our paper and code.

Data
Two examples from the Balanced dataset, a strike (left) and a swim (right).

Our goal is to detect feeding strikes of larval fish (left) in untrimmed videos. To do this, we decided to take an action classification approach to distinguish between swimming (right), and striking behavior (left). We present a total of 3 datasets, one uncurated and two curated:
  1. Uncurated - 11 untrimmed videos: 1080x1092 pixels, 20-30 minutes @ 750fps. Annotation are spatio-temporal locations of the feeding strike events. [Uncurated - data coming soon]
  2. Balanced - A highly curated dataset; clips are cropped tightly around single fish performing a single behavior, lighting is fairly even and no occlusions are present. 71 swim clips and 66 strike clips, clip length varies, average is 46 frames, average size is 3852 pixels. The annotation here is simply "strike" or "swim", clips are divided into the train/val/test. The two examples above are from this dataset. [Balanced]
  3. Naturalistic - resulted from our curated experiment (see below); clips are more challenging in terms of lighting and occlusions. There are 63 strike clips and 4,500 other clips:, 118 contain no fish and 4,382 contain mostly swimming fish. Clip length is 80 frames, size varies between 250-6502 pixels. The annotation here is "strike" and everything else is considered a "swim". [Naturalistic]
Two examples from the Naturalistic dataset, a strike (left) and a swim (right).

[Uncurated - soon]
[Balanced]
[Naturalistic]

Code
Visualization of our analysis pipeline and the curated experiment.

To detect feeding strikes of larval fish we take a detection followed by action classification approach. We trained a Faster-R-CNN using Detectron2 to detect our fish. We also trained a SlowFast action classifier using PyTorchVideo to classify the fish's behavior into one of two classes 'swim' or 'strike'.
We test this pipeline in a curated experiment - we sample the frames where we know strike occurs, and additional frames where we know strikes do not occur. To simulate a more natural ratio of events we sample roughly ~10 times more frames without strikes. For each frame sampled we applied the detector, followed by the action classifier. By doing this for the 11 videos from the uncurated dataset with the highest strike rates we created the Naturalistic dataset.
The code for training the SlowFast model, including some custom data augmentations, and for this experiment will be available in our repository soon.

 [GitHub]

Performance evaluation
Results from our simulation study showing the performance for the performance of the same classifier varies greatly under different data imbalances.
The fact that our events of interested are extremely rare results in a very small sample size and high data imbalance. ROCs are known to be overly optimistic for imbalanced data. PRCs are a popular alternative but it is less known that they are sensitive to the level of data imbalance in data, even the same classifier can produce dramatically different results. This is crucial when taking a trained classifier out to the wild, where feeding strikes are much more sporadic than in our curated datasets.
We developed a simulation scheme that enables us to assess what performance might be under small sample size at different levels of data imbalance and with different classifier qualities. This is an R code utilizing the precrec for ROC and PRC calculations, you can find it in our repository.

Paper and Supplementary Material

Pre-print on biorxiv.


Questions?

We'd love to hear them! Feel free to contact the corresponding author at shirbar@tauex.tau.ac.il


Acknowledgements

This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.