Context-based Virtual Adversarial Training for Text Classification with Noisy Labels

  title={Context-based Virtual Adversarial Training for Text Classification with Noisy Labels},
  author={Do-Myoung Lee and Yeachan Kim and Chang-gyun Seo},
Deep neural networks (DNNs) have a high capacity to completely memorize noisy labels given sufficient training time, and its memorization unfortunately leads to performance degradation. Recently, virtual adversarial training (VAT) attracts attention as it could further improve the generalization of DNNs in semi-supervised learning. The driving force behind VAT is to prevent the models from overffiting to data points by enforcing consistency between the inputs and the perturbed inputs. These… 

Figures and Tables from this paper



Virtual Adversarial Training for Semi-Supervised Text Classification

This work extends adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings in a recurrent neural network rather than to the original input itself.

SeqVAT: Virtual Adversarial Training for Semi-Supervised Sequence Labeling

Empirical studies show that SeqVAT not only significantly improves the sequence labeling performance over baselines under supervised settings, but also outperforms state-of-the-art approaches under semi-supervised settings.

Distributional Smoothing with Virtual Adversarial Training

When the LDS based regularization was applied to supervised and semi-supervised learning for the MNIST dataset, it outperformed all the training methods other than the current state of the art method, which is based on a highly advanced generative model.

How does Disagreement Help Generalization against Label Corruption?

A robust learning paradigm called Co-teaching+, which bridges the "Update by Disagreement" strategy with the original Co-Teaching, which is much superior to many state-of-the-art methods in the robustness of trained models.

An Effective Label Noise Model for DNN Text Classification

This approach introduces a non-linear processing layer (noise model) that models the statistics of the label noise into a convolutional neural network (CNN) architecture and enables the CNN to learn better sentence representations and is robust even to extreme label noise.

Co-teaching: Robust training of deep neural networks with extremely noisy labels

Empirical results on noisy versions of MNIST, CIFar-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.

MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels

Experimental results demonstrate that the proposed novel technique of learning another neural network, called MentorNet, to supervise the training of the base deep networks, namely, StudentNet, can significantly improve the generalization performance of deep networks trained on corrupted training data.

Combating Noisy Labels by Agreement: A Joint Training Method with Co-Regularization

A robust learning paradigm called JoCoR is proposed, which aims to reduce the diversity of two networks during training by using two networks to make predictions on the same mini-batch data and calculate a joint loss with Co-Regularization for each training example.

Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks

The proposed novel framework for training deep convolutional neural networks from noisy labeled datasets that can be obtained cheaply is applied to the image labeling problem and is shown to be effective in labeling unseen images as well as reducing label noise in training on CIFAR-10 and MS COCO datasets.

CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise

CleanNet, a joint neural embedding network, which only requires a fraction of the classes being manually verified to provide the knowledge of label noise that can be transferred to other classes is introduced, which can reduce label noise detection error rate on held-out classes where no human supervision available.