DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

@article{Mathis2018DeepLabCutMP,
  title={DeepLabCut: markerless pose estimation of user-defined body parts with deep learning},
  author={Alexander Mathis and Pranav Mamidanna and Kevin M. Cury and Taiga Abe and Venkatesh N. Murthy and M. Mathis and Matthias Bethge},
  journal={Nature Neuroscience},
  year={2018},
  volume={21},
  pages={1281-1289}
}
Quantifying behavior is crucial for many applications in neuroscience. [] Key Result We demonstrate the versatility of this framework by tracking various body parts in multiple species across a broad collection of behaviors. Remarkably, even when only a small number of frames are labeled (~200), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy.Using a deep learning approach to track user-defined body parts during various behaviors across multiple…
Using DeepLabCut for 3D markerless pose estimation across species and behaviors
TLDR
An updated toolbox that is self contained within a Python package that includes new features such as graphical user interfaces and active-learning based network refinement is provided.
Large-scale capture of hidden fluorescent labels for training generalizable markerless motion capture models
TLDR
GlowTrack is developed, an automated, high-throughput approach for generating large training datasets that overcome the relatively modest limits of manual labeling, enabling deep learning models that generalize across experimental contexts.
Deep-Learning for Automated Markerless Tracking of Infants General Movements
TLDR
A robust markerless pose-estimation scheme, based on advanced deep-learning technology, to track infant movements in consumer mobile device video recordings, indicating the possibility of establishing an automated GM tracking platform, as a suitable alternative to, or support for, the current observational protocols for early diagnosis of neurodevelopmental disorders in early infancy.
TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields
TLDR
TRex is presented, a fast and easy-to-use solution for tracking a large number of individuals simultaneously with real-time (60Hz) tracking performance for up to approximately 256 individuals and estimates 2D body postures and visual fields, both in open and closed-loop contexts.
DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
TLDR
A new easy-to-use software toolkit, DeepPoseKit, is introduced that addresses animal pose estimation problems using an eZcient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision.
A Review of Deep Learning Techniques for Markerless Human Motion on Synthetic Datasets
TLDR
Based on powerful deep learning techniques and recently collected real-world datasets, this work explored a model that can predict the skeleton of an animation based solely on 2D images and produces an animated skeleton for human movement.
Quantifying the kinematic features of dexterous finger movements in nonhuman primates with markerless tracking
TLDR
DeepLabCut, an open-source deep learning toolset, is explored in combination with a standard behavioral task to quantify nonhuman primate performance in precision grasping to provide objective quantitative metrics and crucial information for assessing movement impairments across populations and the potential translation of treatments, interventions and their outcomes.
The accuracy of several pose estimation methods for 3D joint centre localisation
TLDR
3D joint centre locations are not yet consistently comparable to marker-based motion capture but pose estimation derived joint centres demonstrated systematic differences at the hip and knee, most likely due to mislabeling of ground truth data in the training datasets.
Anipose: A toolkit for robust markerless 3D pose estimation
TLDR
This work introduces Anipose, a Python toolkit for robust markerless 3D pose estimation and hopes this open-source software and accompanying tutorials will facilitate the analysis of 3D animal behavior and the biology that underlies it.
...
...

References

SHOWING 1-10 OF 112 REFERENCES
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
TLDR
A novel benchmark "MPII Human Pose" is introduced that makes a significant advance in terms of diversity and difficulty, a contribution that is required for future developments in human body models.
ArtTrack: Articulated Multi-Person Tracking in the Wild
TLDR
This paper uses a model that resembles existing architectures for single-frame pose estimation but is substantially faster to generate proposals for body joint locations and forms articulated tracking as spatio-temporal grouping of such proposals.
DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model
The goal of this paper is to advance the state-of-the-art of articulated pose estimation in scenes with multiple people. To that end we contribute on three fronts. We propose (1) improved body part
DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation
TLDR
An approach that jointly solves the tasks of detection and pose estimation: it infers the number of persons in a scene, identifies occluded body parts, and disambiguates body parts between people in close proximity of each other is proposed.
Superpixels based marker tracking vs. hue thresholding in rodent biomechanics application
TLDR
This work proposes two methods to segment and track mice and rats using SLIC superpixels segmentation with a tracker based on position, speed, shape, and color information of the segmented region in the previous frame; second, using a thresholding on hue channel following up with the same tracker.
A Markerless 3D Computerized Motion Capture System Incorporating a Skeleton Model for Monkeys
TLDR
The results suggest that the presented MCS in monkeys is useful in investigating neural mechanisms underlying various psychiatric disorders and developing pharmacological interventions.
FlyLimbTracker: An active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila
TLDR
A method that uses active contours to semi-automatically track body and leg segments from video image sequences of unmarked, freely behaving Drosophila is described and its open-source software implementation, FlyLimbTracker, is provided, opening up the possibility of tracking leg movements in other species by modifications of underlying active contour models.
Convolutional Pose Machines
TLDR
This work designs a sequential architecture composed of convolutional networks that directly operate on belief maps from previous stages, producing increasingly refined estimates for part locations, without the need for explicit graphical model-style inference in structured prediction tasks such as articulated pose estimation.
Cascaded pose regression
TLDR
Experiments suggest CPR is fast (2–3ms per pose estimate), accurate (approaching human performance), and easy to train from small amounts of labeled data.
DeepPose: Human Pose Estimation via Deep Neural Networks
TLDR
The pose estimation is formulated as a DNN-based regression problem towards body joints and a cascade of such DNN regres- sors which results in high precision pose estimates.
...
...