• Publications
  • Influence
Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
TLDR
It is shown that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora and carries the further advantages of improved factual accuracy and reduced toxicity in the resulting language model.
End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems
TLDR
This model comprises a single transformer-based encoder-decoder network that is trained end-to-end to generate both answers and questions and indicates significant improvements in the domain adaptation of QA models outperforming current state-of-the-art methods.
Differentiable Greedy Networks
TLDR
This paper proposes a subset selection algorithm that is trainable with gradient-based methods yet achieves near-optimal performance via submodular optimization, and shows that the proposed differentiable greedy network (DGN) outperforms discrete optimization algorithms as well as other baseline methods in terms of precision and recall.
Towards Zero-Shot Multilingual Synthetic Question and Answer Generation for Cross-Lingual Reading Comprehension
TLDR
This work proposes a simple method to generate large amounts of multilingual question and answer pairs by a single generative model, thus removing the need for human annotations in the target languages.
Embedding-based Zero-shot Retrieval through Query Generation
TLDR
This work considers the embedding-based two-tower architecture as the neural retrieval model and proposes a novel method for generating synthetic training data for retrieval, which produces remarkable results, significantly outperforming BM25 on 5 out of 6 datasets tested.
ParsiNLU: A Suite of Language Understanding Challenges for Persian
TLDR
This work introduces ParsiNLU, the first benchmark in Persian language that includes a range of language understanding tasks—reading comprehension, textual entailment, and so on, and presents the first results on state-of-the-art monolingual and multilingual pre-trained language models on this benchmark and compares them with human performance.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
TLDR
Evaluation of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters finds that model performance and calibration both improve with scale, but are poor in absolute terms.
EncT5: Fine-tuning T5 Encoder for Non-autoregressive Tasks
TLDR
This work proposes EncT5 as a way to efficiently fine-tune pre-trained encoder-decoder T5 models for classification and regression tasks by using the encoder layers.
TableQnA: Answering List Intent Queries With Web Tables
TLDR
The authors' experiments on real-life web search queries show that the intent extractor for list and superlative intent queries has significantly higher precision and coverage compared with baseline approaches and the table answer selector significantly outperforms the state-of-the-art baseline approach.
Open Domain Question Answering Using Web Tables
TLDR
The key insight is to combine deep neural network-based semantic similarity between the query and the table with features that quantify the dominance of the table in the document as well as the quality of the information in the table.
...
...