• Publications
  • Influence
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
TLDR
After pretraining on large-scale image-caption pairs, Unicoder-VL is transferred to caption-based image-text retrieval and visual commonsense reasoning, with just one additional output layer, and the powerful ability of the cross-modal pre-training is shown. Expand
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
TLDR
This work develops CodeBERT with Transformer-based neural architecture, and trains it with a hybrid objective function that incorporates the pre-training task of replaced token detection, which is to detect plausible alternatives sampled from generators. Expand
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
TLDR
A new sequence-to-sequence pre-training model called ProphetNet is presented, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism that predicts the next n tokens simultaneously based on previous context tokens at each time step. Expand
Constraint-Based Question Answering with Knowledge Graph
TLDR
A novel systematic KBQA approach to solve multi-constraint questions is proposed, which not only obtains comparable results on the two existing benchmark data-sets, but also achieves significant improvements on the ComplexQuestions. Expand
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
TLDR
K-Adapter is proposed, which remains the original parameters of the pre-trained model fixed and supports continual knowledge infusion and captures richer factual and commonsense knowledge than RoBERTa. Expand
Pretraining-Based Natural Language Generation for Text Summarization
TLDR
A novel pretraining-based encoder-decoder framework, which can generate the output sequence based on the input sequence in a two-stage manner, which achieves new state-of-the-art on both CNN/Daily Mail and New York Times datasets. Expand
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
TLDR
It is found that doing fine-tuning on multiple languages together can bring further improvement in Unicoder, a universal language encoder that is insensitive to different languages. Expand
Question Generation for Question Answering
TLDR
Experimental results show that, by using generated questions as an extra signal, significant QA improvement can be achieved. Expand
Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering
TLDR
This work proposes to automatically extract evidence from heterogeneous knowledge sources, and answer questions based on the extracted evidence, and achieves the state-of-the-art accuracy on the CommonsenseQA dataset. Expand
Reasoning Over Semantic-Level Graph for Fact Checking
TLDR
This work proposes two mechanisms to exploit the structure of evidence while leveraging the advances of pre-trained models like BERT, GPT or XLNet, and is the state-of-the-art system in terms of both official evaluation metrics, namely claim verification accuracy and FEVER score. Expand
...
1
2
3
4
5
...