DeepLabCut: markerless pose estimation of user-defined body parts with deep learning

@article{Mathis2018DeepLabCutMP,
  title={DeepLabCut: markerless pose estimation of user-defined body parts with deep learning},
  author={Alexander Mathis and Pranav Mamidanna and Kevin M. Cury and Taiga Abe and V. Murthy and M. Mathis and M. Bethge},
  journal={Nature Neuroscience},
  year={2018},
  volume={21},
  pages={1281-1289}
}
Quantifying behavior is crucial for many applications in neuroscience. [...] Key Result We demonstrate the versatility of this framework by tracking various body parts in multiple species across a broad collection of behaviors. Remarkably, even when only a small number of frames are labeled (~200), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy.Using a deep learning approach to track user-defined body parts during various behaviors across multiple…Expand

Paper Mentions

Observational Clinical Trial
The objective of this study is the development of a system that will allow for the precise measurement of movement kinematics in a clinical exam setting using natural video from three… Expand
ConditionsMovement Disorders
Using DeepLabCut for 3D markerless pose estimation across species and behaviors
TLDR
This protocol describes how to use an open-source toolbox, DeepLabCut, to train a deep neural network to precisely track user-defined features with limited training data, which allows noninvasive behavioral tracking of movement. Expand
Using DeepLabCut for 3D markerless pose estimation across species and behaviors
TLDR
An updated toolbox that is self contained within a Python package that includes new features such as graphical user interfaces and active-learning based network refinement is provided. Expand
Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio
TLDR
OpenMonkeyStudio, a deep learning-based markerless motion capture system for estimating 3D pose in freely moving macaques in large unconstrained environments, and it is shown that OpenMonkey studio can be used to accurately recognize actions and track social interactions. Expand
SLEAP: Multi-animal pose tracking
TLDR
SLEAP (Social LEAP Estimates Animal Poses), a framework for multi-animal pose tracking via deep learning, is presented, capable of simultaneously tracking any number of animals during social interactions and across a variety of experimental conditions. Expand
Fast animal pose estimation using deep neural networks
TLDR
This work validated LEAP using videos of freely behaving fruit flies and tracked 32 distinct points to describe the pose of the head, body, wings and legs, with an error rate of<3% of body length and demonstrated LEAP’s applicability for unsupervised behavioral classification. Expand
DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
TLDR
A new easy-to-use software toolkit, DeepPoseKit, is introduced that addresses animal pose estimation problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. Expand
DeepPoseKit: a software toolkit for fast and robust animal pose estimation using deep learning
TLDR
A new easy-to-use software toolkit, DeepPoseKit, is introduced that addresses animal pose estimation problems using an efficient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision. Expand
3D reconstruction toolbox for behavior tracked with multiple cameras
TLDR
A recently developed toolbox called DeepLabCut (DLC) leverages Artificial Neural Network (ANN) based computer vision to make precise markerless tracking possible for scientific experiments. Expand
TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields
TLDR
TRex is presented, a fast and easy-to-use solution for tracking a large number of individuals simultaneously using background-subtraction with real-time (60Hz) tracking performance for up to approximately 256 individuals and estimates 2D visual-fields, outlines, and head/rear of bilateral animals, both in open and closed-loop contexts. Expand
OptiFlex: Multi-Frame Animal Pose Estimation Combining Deep Learning With Optical Flow
TLDR
This work introduces a novel multi-frame animal pose estimation framework, referred to as OptiFlex, which integrates a flexible base model, which accounts for variability in animal body shape, with an OpticalFlow model that incorporates temporal context from nearby video frames. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 90 REFERENCES
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
TLDR
A novel benchmark "MPII Human Pose" is introduced that makes a significant advance in terms of diversity and difficulty, a contribution that is required for future developments in human body models. Expand
ArtTrack: Articulated Multi-Person Tracking in the Wild
TLDR
This paper uses a model that resembles existing architectures for single-frame pose estimation but is substantially faster to generate proposals for body joint locations and forms articulated tracking as spatio-temporal grouping of such proposals. Expand
DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model
The goal of this paper is to advance the state-of-the-art of articulated pose estimation in scenes with multiple people. To that end we contribute on three fronts. We propose (1) improved body partExpand
DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation
TLDR
An approach that jointly solves the tasks of detection and pose estimation: it infers the number of persons in a scene, identifies occluded body parts, and disambiguates body parts between people in close proximity of each other is proposed. Expand
Superpixels based marker tracking vs. hue thresholding in rodent biomechanics application
TLDR
This work proposes two methods to segment and track mice and rats using SLIC superpixels segmentation with a tracker based on position, speed, shape, and color information of the segmented region in the previous frame; second, using a thresholding on hue channel following up with the same tracker. Expand
A Markerless 3D Computerized Motion Capture System Incorporating a Skeleton Model for Monkeys
TLDR
The results suggest that the presented MCS in monkeys is useful in investigating neural mechanisms underlying various psychiatric disorders and developing pharmacological interventions. Expand
FlyLimbTracker: an active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila
TLDR
A method that uses active contours to semi-automatically track body and leg segments from video image sequences of unmarked, freely behaving Drosophila is described and its open-source software implementation, FlyLimbTracker, is provided, opening up the possibility of tracking leg movements in other species by modifications of underlying active contour models. Expand
Convolutional Pose Machines
TLDR
This work designs a sequential architecture composed of convolutional networks that directly operate on belief maps from previous stages, producing increasingly refined estimates for part locations, without the need for explicit graphical model-style inference in structured prediction tasks such as articulated pose estimation. Expand
Cascaded pose regression
TLDR
Experiments suggest CPR is fast (2–3ms per pose estimate), accurate (approaching human performance), and easy to train from small amounts of labeled data. Expand
DeepPose: Human Pose Estimation via Deep Neural Networks
TLDR
The pose estimation is formulated as a DNN-based regression problem towards body joints and a cascade of such DNN regres- sors which results in high precision pose estimates. Expand
...
1
2
3
4
5
...