Skip to search formSkip to main contentSkip to account menu

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being…
View PDF on arXiv (opens in a new tab)

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks and supports distributed training across multiple GPUs and machines.
View on ACL (opens in a new tab)

wav2vec: Unsupervised Pre-training for Speech Recognition

Wav2vec is trained on large amounts of unlabeled audio data and the resulting representations are then used to improve acoustic model training and outperforms Deep Speech 2, the best reported character-based system in the literature while using two orders of magnitude less labeled training data.
View PDF on arXiv (opens in a new tab)

Pay Less Attention with Lightweight and Dynamic Convolutions

It is shown that a very lightweight convolution can perform competitively to the best reported self-attention results, and dynamic convolutions are introduced which are simpler and more efficient than self-ATTention.
View PDF on arXiv (opens in a new tab)

Unsupervised Cross-lingual Representation Learning for Speech Recognition

XLSR is presented which learns cross-lingual speech representations by pretraining a single model from the raw waveform of speech in multiple languages to enable a single multilingual speech recognition model which is competitive to strong individual models.
View PDF on arXiv (opens in a new tab)

vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations

Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition and the algorithm uses a gumbel softmax or online k-means clustering to quantize the dense representations.
View PDF on arXiv (opens in a new tab)

Facebook FAIR’s WMT19 News Translation Task Submission

This paper describes Facebook FAIR’s submission to the WMT19 shared news translation task and achieves the best case-sensitive BLEU score for the translation direction English→Russian.
View on ACL (opens in a new tab)

Adaptive Input Representations for Neural Language Modeling

Adapt input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity are introduced and a systematic comparison of popular choices for a self-attentional architecture is performed.
View PDF on arXiv (opens in a new tab)

Unsupervised Speech Recognition

Compared to the best previous unsupervised work, wav2vec-U reduces the phoneme error rate on the TIMIT benchmark from 26.1 to 11.3 and rivaling some of the best published systems trained on 960 hours of labeled data from only two years ago.
View PDF on arXiv (opens in a new tab)

Cloze-driven Pretraining of Self-attention Networks

A new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems, including cloze-style word reconstruction task, and a detailed analysis of a number of factors that contribute to effective pretraining.
View on ACL (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)