Peer review process
Revised: This Reviewed Preprint has been revised by the authors in response to the previous round of peer review; the eLife assessment and the public reviews have been updated where necessary by the editors and peer reviewers.
Read more about eLife’s peer review process.Editors
- Reviewing EditorJenny TungMax Planck Institute for Evolutionary Anthropology, Leipzig, Germany
- Senior EditorGeorge PerryPennsylvania State University, University Park, United States of America
Reviewer #1 (Public review):
Summary:
Advances in machine vision and computer learning have meant that there are now state-of-the-art and open-source toolboxes that allow for animal pose estimation and action recognition. These technologies have the potential to revolutionize behavioral observations of wild primates but are often held back by labor intensive model training and the need for some programming knowledge to effectively leverage such tools. The study presented here by Fuchs et al unveils a new framework (ASBAR) that aims to automate behavioral recognition in wild apes from video data. This framework combines robustly trained and well tested pose estimate and behavioral action recognition models. The framework performs admirably at the task of automatically identifying simple behaviors of wild apes from camera trap videos of variable quality and contexts. These results indicate that skeletal-based action recognition offers a reliable and lightweight methodology for studying ape behavior in the wild and the presented framework and GUI offer an accessible route for other researchers to utilize such tools.
Given that automated behavior recognition in wild primates will likely be a major future direction within many subfields of primatology, open-source frameworks, like the one presented here, will present a significant impact on the field and will provide a strong foundation for others to build future research upon.
Strengths:
Clearly articulated the argument as to why the framework was needed and what advantages it could convey to the wider field.
For a very technical paper it was very well written. Every aspect of the framework the authors clearly explained why it was chosen and how it was trained and tested. This information was broken down in a clear and easily digestible way that will be appreciated by technical and non-technical audiences alike.
The study demonstrates which pose estimation architectures produce the most robust models for both within context and out of context pose estimates. This is invaluable knowledge for those wanting to produce their own robust models.
The comparison of skeletal-based action recognition with other methodologies for action recognition are helpful in contextualizing the results.
Weaknesses:
While I note that this is a paper most likely aimed at the more technical reader, it will also be of interest to a wider primatological readership, including those who work extensively in the field. When outlining the need for future work I felt the paper offered almost exclusively very technical directions. This may have been a missed opportunity to engage the wider readership and suggest some practical ways those in the field could collect more ASBAR friendly video data to further improve accuracy.
Comments on latest version:
I think the new version is an improvement and applaud the authors on a well-written article that conveys some very technical details excellently. The authors have addressed my initial comments about reaching out to a wider, sometimes less technical, primatological audience by encouraging researchers to create large annotated datasets and make these publicly accessible. I also agree that fostering interdisciplinary collaboration is the best way to progress this field of research. These additions have certainly strengthened the paper but I still think some more practical advice for the actual collection of high-quality training data used to improve the pose estimates and behavioral classification in tough out-of-context environments could have been added. This doesn't detract from the quality of the paper though.
Reviewer #2 (Public review):
Fuchs et al. propose a framework for action recognition based on pose estimation. They integrate functions from DeepLabCut and MMAction2, two popular machine learning frameworks for behavioral analysis, in a new package called ASBAR.
They test their framework by:
Running pose estimation experiments on the OpenMonkeyChallenge (OMC) dataset (the public train + val parts) with DeepLabCut
Also annotating around 320 images pose data in the PanAf dataset (which contains behavioral annotations). They show that the ResNet-152 model generalizes best from the OMC data to this out-of-domain dataset.
They then train a skeleton-based action recognition model on PanAf and show that the top-1/3 accuracy is slightly higher than video-based methods