Corpus ID: 4554201

AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video

@inproceedings{Wang2018AJILEMP,
  title={AJILE Movement Prediction: Multimodal Deep Learning for Natural Human Neural Recordings and Video},
  author={X. Wang and Ali Farhadi and Rajesh P. N. Rao and Bingni W. Brunton},
  booktitle={AAAI},
  year={2018}
}
Developing useful interfaces between brains and machines is a grand challenge of neuroengineering. An effective interface has the capacity to not only interpret neural signals, but predict the intentions of the human to perform an action in the near future; prediction is made even more challenging outside well-controlled laboratory experiments. This paper describes our approach to detect and to predict natural human arm movements in the future, a key challenge in brain computer interfacing that… Expand
AJILE12: Long-term naturalistic human intracranial neural recordings and pose
TLDR
The Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset is described, the largest human neurobehavioral dataset that is publicly available; the dataset was recorded opportunistically during passive clinical epilepsy monitoring. Expand
Mining naturalistic human behaviors in long-term video and neural recordings
TLDR
An automated, behavior-first approach for analyzing simultaneously recorded long-term, naturalistic electrocorticography (ECoG) and behavior video data and contributes methods that may generalize to other neural recording modalities beyond ECoG. Expand
Towards naturalistic human neuroscience and neuroengineering: behavior mining in long-term video and neural recordings
TLDR
An automated approach for analyzing large datasets of simultaneously recorded human electrocorticography (ECoG) and naturalistic behavior video data for 12 subjects and plans to publish a curated dataset, which captures naturalistic neural and behavioral variability at a scale not previously available. Expand
Enabling naturalistic neuroscience through behavior mining: Analysis of long-term human brain and video recordings
Much of our understanding in human neuroscience has been informed by data collected in pre-designed and wellcontrolled experimental tasks, where timings of cues, stimuli, and behavioral responses areExpand
Behavioral and Neural Variability of Naturalistic Arm Movements
TLDR
This work is among the first studies connecting behavioral and neural variability across cortex in humans during unstructured movements and contributes to the understanding of long-term naturalistic behavior. Expand
Behavioral and Neural Variability of Naturalistic Arm Movements
TLDR
This work leveraged recent advances in machine learning and computer vision to analyze intracranial recordings from 12 human subjects during thousands of spontaneous, unstructured arm reach movements, observed over several days for each subject. Expand
Stimulus Speech Decoding from Human Cortex with Generative Adversarial Network Transfer Learning
TLDR
A transfer learning approach with a pre-trained GAN to disentangle representation and generation layers for decoding and demonstrates a high reconstruction accuracy using deep learning networks together with the potential to elucidate interactions across different brain regions during a cognitive task. Expand
Learning neural decoders without labels using multiple data streams
TLDR
This work learns neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals by applying cross-modal, self-supervised deep clustering to decode movements from brain recordings. Expand
Intracranial Error Detection via Deep Learning
  • M. Völker, J. Hammer, +6 authors T. Ball
  • Computer Science, Biology
  • 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
  • 2018
TLDR
The usefulness of deep learning is shown for both intracranial error decoding and mapping of the spatio-temporal structure of the human error processing network. Expand
Autoencoders for learning template spectrograms in electrocorticographic signals.
TLDR
An autoencoder based approach for summarizing ECoG data with 'template spectrograms', i.e. informative time-frequency (t-f) patterns, and an unsupervised method for clustering electrode channels based on their aggregated activity are proposed and evaluated. Expand
...
1
2
3
...

References

SHOWING 1-10 OF 50 REFERENCES
Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations
TLDR
This work analyzed continuous, long-term electrocorticography data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings to describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. Expand
Neural correlates to automatic behavior estimations from RGB-D video in epilepsy unit
TLDR
This work presents a functional map of electrode responses to natural arm movements, generated using a statistical test, and demonstrates coarse functional mapping of ECoG electrodes correlated to contralateral arm movements. Expand
Deep learning with convolutional neural networks for brain mapping and decoding of movement-related information from the human EEG
TLDR
This study shows how to design and train ConvNets to decode movement-related information from the raw EEG without handcrafted features and highlights the potential of deep convolutional neural networks combined with advanced visualization techniques for EEG-based brain mapping. Expand
Prediction of Three-Dimensional Arm Trajectories Based on ECoG Signals Recorded from Human Sensorimotor Cortex
TLDR
3D arm trajectories in time series from ECoG signals in humans are predicted using a novel preprocessing method and a sparse linear regression, confirming the feasibility of predicting 3D arm trajectory reconstruction for a human limb from E coG. Expand
Flowing ConvNets for Human Pose Estimation in Videos
TLDR
This work proposes a ConvNet architecture that is able to benefit from temporal context by combining information across the multiple frames using optical flow and outperforms a number of others, including one that uses optical flow solely at the input layers, one that regresses joint coordinates directly, and one that predicts heatmaps without spatial fusion. Expand
Deep learning with convolutional neural networks for EEG decoding and visualization
TLDR
This study shows how to design and train convolutional neural networks to decode task‐related information from the raw EEG without handcrafted features and highlights the potential of deep ConvNets combined with advanced visualization techniques for EEG‐based brain mapping. Expand
Deep Feature Learning Using Target Priors with Applications in ECoG Signal Decoding for BCI
TLDR
This work describes a weakly supervised learning method of a prior supervised convolutional stacked auto-encoders (PCSA), of which information in the target variables is represented probabilistically using a Gaussian Bernoulli restricted Boltzmann machine (RBM). Expand
Using the electrocorticographic speech network to control a brain-computer interface in humans.
TLDR
It is demonstrated here for the first time that ECoG signals associated with different overt and imagined phoneme articulation can enable invasively monitored human patients to control a one-dimensional computer cursor rapidly and accurately. Expand
Decoding natural grasp types from human ECoG
TLDR
It is shown that two different grasp types can be reliably distinguished in natural reach-to-grasp movements in single-trial ECoG recordings from the human motor cortex, and it is concluded that E coG signals are promising candidates for BMIs that include the restoration of grasping movements. Expand
Methods Towards Invasive Human Brain Computer Interfaces
TLDR
This paper presents the method and examples of intracranial EEG recordings of three epilepsy patients with electrode grids placed on the motor cortex, and analyzes the classifiability of the data using Support Vector Machines (SVMs) and Recursive Channel Elimination (RCE). Expand
...
1
2
3
4
5
...