Variational Pretraining for Semi-supervised Text Classification

@inproceedings{Gururangan2019VariationalPF,
  title={Variational Pretraining for Semi-supervised Text Classification},
  author={Suchin Gururangan and T. Dang and D. Card and Noah A. Smith},
  booktitle={ACL},
  year={2019}
}
  • Suchin Gururangan, T. Dang, +1 author Noah A. Smith
  • Published in ACL 2019
  • Computer Science
  • We introduce VAMPIRE, a lightweight pretraining framework for effective text classification when data and computing resources are limited. We pretrain a unigram document model as a variational autoencoder on in-domain, unlabeled data and use its internal states as features in a downstream classifier. Empirically, we show the relative strength of VAMPIRE against computationally expensive contextual embeddings and other popular semi-supervised baselines under low resource settings. We also find… CONTINUE READING
    19 Citations
    MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification
    • 8
    • Highly Influenced
    • PDF
    Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
    • 11
    • PDF
    Discrete Latent Variable Representations for Low-Resource Text Classification
    • 1
    • Highly Influenced
    • PDF
    Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation
    • 34
    • PDF
    A Discrete Variational Recurrent Topic Model without the Reparametrization Trick

    References

    SHOWING 1-10 OF 45 REFERENCES
    Universal Language Model Fine-tuning for Text Classification
    • 1,237
    • PDF
    Virtual Adversarial Training for Semi-Supervised Text Classification
    • 61
    • PDF
    Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
    • 118
    • PDF
    Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
    • 207
    • Highly Influential
    • PDF
    Variational Autoencoder for Semi-Supervised Text Classification
    • 123
    • Highly Influential
    • PDF
    Deep Unordered Composition Rivals Syntactic Methods for Text Classification
    • 518
    • PDF
    Semi-supervised Sequence Learning
    • 650
    • PDF
    Improving Language Understanding by Generative Pre-Training
    • 1,528
    • PDF
    Semi-supervised Learning with Deep Generative Models
    • 1,520
    • PDF
    Dissecting Contextual Word Embeddings: Architecture and Representation
    • 162
    • Highly Influential
    • PDF