Overcoming the Domain Gap in Neural Action Representations

  title={Overcoming the Domain Gap in Neural Action Representations},
  author={Semih Gunel and Florian Aymanns and Sina Honari and Pavan Ramdya and P. Fua},
Relating behavior to brain activity in animals is a fundamental goal in neuroscience, with practical applications in building robust brain-machine interfaces. However, the domain gap between individuals is a major issue that prevents the training of general models that work on unlabeled subjects. Since 3D pose data can now be reliably extracted from multi-view video sequences without manual intervention, we propose to use it to guide the encoding of neural action representations together with a… 



AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video

A multimodal model that combines deep convolutional neural networks (CNN) with long short-term memory (LSTM) blocks, leveraging both ECoG and video modalities is proposed, and it is demonstrated that the models are able to detect movements and predict future movements up to 800 msec before movement initiation.

Learning neural decoders without labels using multiple data streams

It is found that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data.

Adversarial Domain Adaptation for Stable Brain-Machine Interfaces

Brain-Machine Interfaces (BMIs) have recently emerged as a clinically viable option to restore voluntary movements after paralysis. These devices are based on the ability to extract information about

BehaveNet: nonlinear embedding and Bayesian neural decoding of behavioral videos

This work introduces a probabilistic framework for the analysis of behavioral video and neural activity, which provides tools for compression, segmentation, generation, and decoding of behavioral videos and develops a novel Bayesian decoding approach.

Task Programming: Learning Data Efficient Behavior Representations

TREBA is presented: a method to learn annotation-sample efficient trajectory embedding for behavior analysis, based on multi-task self-supervised learning and task programming, which uses programs to explicitly encode structured knowledge from domain experts.

DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning

A new easy-to-use software toolkit, DeepPoseKit, is introduced that addresses animal pose estimation problems using an eZcient multi-scale deep-learning model, called Stacked DenseNet, and a fast GPU-based peak-detection algorithm for estimating keypoint locations with subpixel precision.

Multi-source domain adaptation for decoder calibration of intracortical brain-machine interface

A principal component analysis (PCA)-based multi-source domain adaptation (PMDA) algorithm was proposed to apply the feature transfer to diminish the disparities between the target domain and each source domain, resulting in better and more robust decoding performance.

Deformation-Aware Unpaired Image Translation for Pose Estimation on Laboratory Animals

A new sim2real domain transfer method for explicit and independent modeling of appearance, shape and pose in an unpaired image translation framework for neuroscience model organisms to be able to study how neural circuits orchestrate behaviour.

SLEAP: Multi-animal pose tracking

SLEAP (Social LEAP Estimates Animal Poses), a framework for multi-animal pose tracking via deep learning, is presented, capable of simultaneously tracking any number of animals during social interactions and across a variety of experimental conditions.