Skip to search formSkip to main contentSkip to account menu

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

A new Q\&A architecture called QANet is proposed, which does not require recurrent networks, and its encoder consists exclusively of convolution and self-attention, where convolution models local interactions andSelf-att attention models global interactions.
View PDF on arXiv (opens in a new tab)

Finetuned Language Models Are Zero-Shot Learners

It is shown that instruction tuning —finetuning language models on a collection of datasets described via instructions—substantially improves zero-shot performance on unseen tasks and outperforms few-shot GPT-3 by a large margin on ANLI, RTE, BoolQ, AI2-ARC, OpenbookQA, and StoryCloze.
View PDF on arXiv (opens in a new tab)

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

(b)). These results suggest zero-shot cross-modality transfer emerges with the scaling of weakly labeled data.
View PDF on arXiv (opens in a new tab)

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

It is shown that the orthogonal rectangular matrix can stabilize the distribution of network activations and regularize FNNs and be used as an alternative to standard linear module in optimization over Multiple Dependent Stiefel Manifolds.
View PDF on arXiv (opens in a new tab)

Learning to Skim Text

The proposed model is a modified LSTM with jumping, a recurrent network that learns how far to jump after reading a few words of the input text, which is up to 6 times faster than the standard sequential L STM, while maintaining the same or even better accuracy.
View on ACL (opens in a new tab)

Scaling Instruction-Finetuned Language Models

This result shows that instruction and UL2 continued pre-training are complementary compute-efficient methods to improve the performance of language models without increasing model scale.
View PDF on arXiv (opens in a new tab)

Combined Scaling for Zero-shot Transfer Learning

We present a combined scaling method called BASIC that achieves 85.7% top-1 zero-shot accuracy on the ImageNet ILSVRC-2012 validation set, surpassing the best-published zero-shot models – CLIP and…

Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension

The Neural Symbolic Reader (NeRd), which includes a reader and programmer to encode the passage and question, and a programmer, e.g., LSTM, to generate a program that is executed to produce the answer.
View Paper (opens in a new tab)

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

This paper proposes and develops a family of language models named GLaM, which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants.
View PDF on arXiv (opens in a new tab)

AdaDelay: Delay Adaptive Distributed Stochastic Optimization

DStochastic convex optimization algorithms under a delayed gradient model in which server nodes update parameters and worker nodes compute stochastic (sub)gradients are developed, with noticeable improvements for large-scale real datasets with billions of examples and features.
View Paper (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)