Skip to search formSkip to main contentSkip to account menu

Improved training of end-to-end attention models for speech recognition

This work introduces a new pretraining scheme by starting with a high time reduction factor and lowering it during training, which is crucial both for convergence and final performance, and trains long short-term memory (LSTM) language models on subword units.
View PDF on arXiv (opens in a new tab)

RWTH ASR Systems for LibriSpeech: Hybrid vs Attention - w/o Data Augmentation

We present state-of-the-art automatic speech recognition (ASR) systems employing a standard hybrid DNN/HMM architecture compared to an attention-based encoder-decoder design for the LibriSpeech task.…
View PDF on arXiv (opens in a new tab)

Language Modeling with Deep Transformers

The analysis of attention weights shows that deep autoregressive self-attention models can automatically make use of positional information and it is found that removing the positional encoding even slightly improves the performance of these models.
View PDF on arXiv (opens in a new tab)

On Using SpecAugment for End-to-End Speech Translation

This work investigates a simple data augmentation technique, SpecAugment, for end-to-end speech translation by alleviating overfitting to some extent and shows that the method also leads to significant improvements in various data conditions irrespective of the amount of training data.
View on ACL (opens in a new tab)

A Comparison of Transformer and LSTM Encoder Decoder Models for ASR

We present competitive results using a Transformer encoder-decoder-attention model for end-to-end speech recognition needing less training time compared to a similarly performing LSTM model. We…
View on IEEE (opens in a new tab)

A comprehensive study of deep bidirectional LSTM RNNS for acoustic modeling in speech recognition

A pretraining scheme for LSTMs with layer-wise construction of the network showing good improvements especially for deep networks is introduced, and a comparison of computation times vs. recognition performance is compared.
View on IEEE (opens in a new tab)

Towards Online-Recognition with Deep Bidirectional LSTM Acoustic Models

This work applies a modification to bidirectional RNNs to enable online-recognition by moving a window over the input stream and perform one forwarding through the RNN on each window, and shows in experiments that the performance of this online-enabled biddirectional LSTM performs as good as the offline bidirectionals and much better than the unidirectional L STM.
View via Publisher (opens in a new tab)

RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition

It is shown that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks.
View on ACL (opens in a new tab)

The RWTH/UPB/FORTH System Combination for the 4th CHiME Challenge Evaluation

This paper describes automatic speech recognition systems developed jointly by RWTH, UPB and FORTH for the 1ch, 2ch and 6ch track of the 4th CHiME Challenge and compares the ASR performance of different beamforming approaches.

Training Language Models for Long-Span Cross-Sentence Evaluation

This work trains language models based on long short-term memory recurrent neural networks and Transformers using various types of training sequences and studies their robustness with respect to different evaluation modes, showing that models trained with back-propagation over sequences consisting of concatenation of multiple sentences with state carry-over across sequences effectively outperform those trained with the sentence-level training.
View on IEEE (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)