Skip to search formSkip to main contentSkip to account menu

Unified Language Model Pre-training for Natural Language Understanding and Generation

A new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks that compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks.
View PDF on arXiv (opens in a new tab)

Gated Self-Matching Networks for Reading Comprehension and Question Answering

The gated self-matching networks for reading comprehension style question answering, which aims to answer questions from a given passage, are presented and holds the first place on the SQuAD leaderboard for both single and ensemble model.
View on ACL (opens in a new tab)

Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification

Three neural networks are developed to effectively incorporate the supervision from sentiment polarity of text (e.g. sentences or tweets) in their loss functions and the performance of SSWE is improved by concatenating SSWE with existing feature set.
View on ACL (opens in a new tab)

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

This work presents a simple and effective approach to compress large Transformer (Vaswani et al., 2017) based pre-trained models, termed as deep self-attention distillation, and demonstrates that the monolingual model outperforms state-of-the-art baselines in different parameter size of student models.
View PDF on arXiv (opens in a new tab)

Neural Question Generation from Text: A Preliminary Study

A preliminary study on neural question generation from text with the SQuAD dataset is conducted, and the experiment results show that the method can produce fluent and diverse questions.
View PDF on arXiv (opens in a new tab)

Neural Document Summarization by Jointly Learning to Score and Select Sentences

This paper presents a novel end-to-end neural network framework for extractive document summarization by jointly learning to score and select sentences, which significantly outperforms the state-of-the-art extractive summarization models.
View on ACL (opens in a new tab)

Selective Encoding for Abstractive Sentence Summarization

The experimental results show that the proposed selective encoding model outperforms the state-of-the-art baseline models.
View on ACL (opens in a new tab)

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

The experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.
View PDF on arXiv (opens in a new tab)

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

  • Zewen ChiLi Dong M. Zhou
  • Computer Science
    North American Chapter of the Association for…
  • 15 July 2020
An information-theoretic framework that formulates cross-lingual language model pre- training as maximizing mutual information between multilingual-multi-granularity texts is presented and a new pre-training task based on contrastive learning is proposed.
View on ACL (opens in a new tab)

Sentiment Embeddings with Applications to Sentiment Analysis

This work develops a number of neural networks with tailoring loss functions, and applies sentiment embeddings to word-level sentiment analysis, sentence level sentiment classification, and building sentiment lexicons, showing results that consistently outperform context-basedembeddings on several benchmark datasets of these tasks.
View on IEEE (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)