Adversarial Dropout for Recurrent Neural Networks

@article{Park2019AdversarialDF,
  title={Adversarial Dropout for Recurrent Neural Networks},
  author={Sungrae Park and Kyungwoo Song and Mingi Ji and Wonsung Lee and Il-Chul Moon},
  journal={ArXiv},
  year={2019},
  volume={abs/1904.09816}
}
Successful application processing sequential data, such as text and speech, requires an improved generalization performance of recurrent neural networks (RNNs. [] Key Result We demonstrated that minimizing our regularizer improved the effectiveness of the dropout for RNNs on sequential MNIST tasks, semi-supervised text classification tasks, and language modeling tasks.

Figures and Tables from this paper

Survey of Dropout Methods for Deep Neural Networks

The history of dropout methods, their various applications, and current areas of research interest are summarized.

Exploring Dropout Discriminator for Domain Adaptation

Bivariate Beta LSTM

The proposed gate structure enables probabilistic modeling on the gates within the LSTM cell so that the modelers can customize the cell state flow with priors and distributions, and theoretically shows the higher upper bound of the gradient compared to the sigmoid function.

Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction

A new concept, namely confidence threshold (CT), is introduced and the reducing of the confidence threshold, known as confidence threshold reduction (CTR), is proven to improve both the generalization and robustness of the model.

References

SHOWING 1-10 OF 41 REFERENCES

Adversarial Dropout for Supervised and Semi-supervised Learning

Adversarial dropout is a minimal set of dropouts that maximize the divergence between 1) the training supervision and 2) the outputs from the network with the dropouts so that it increases the sparsity of neural networks more than the standard dropout.

Fraternal Dropout

A simple technique called fraternal dropout is proposed that takes advantage of dropout to train two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their (pre-softmax) predictions.

RNNDROP: A novel dropout for RNNS in ASR

Recently, recurrent neural networks (RNN) have achieved the state-of-the-art performance in several applications that deal with temporal data, e.g., speech recognition, handwriting recognition and

Dropout distillation

This work introduces a novel approach, coined "dropout distillation", that allows to train a predictor in a way to better approximate the intractable, but preferable, averaging process, while keeping under control its computational efficiency.

Dropout with Expectation-linear Regularization

This work first formulate dropout as a tractable approximation of some latent variable model, leading to a clean view of parameter sharing and enabling further theoretical analysis, and introduces (approximate) expectation-linear dropout neural networks, whose inference gap the authors are able to formally characterize.

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

This work applies a new variational inference based dropout technique in LSTM and GRU models, which outperforms existing techniques, and to the best of the knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank.

Virtual Adversarial Training for Semi-Supervised Text Classification

This work extends adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself.

Recurrent Dropout without Memory Loss

This paper proposes to drop neurons directly in recurrent connections in a way that does not cause loss of long-term memory, and demonstrates its effectiveness for the most effective modern recurrent network – Long Short-Term Memory network.

The Limitations of Deep Learning in Adversarial Settings

This work formalizes the space of adversaries against deep neural networks (DNNs) and introduces a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs.

Adversarial Training Methods for Semi-Supervised Text Classification

This work extends adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself.