Skip to search formSkip to main contentSkip to account menu
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
TLDR
This work presents the Pile, an 825 GiB English text corpus tar-geted at training large-scale language models, constructed from 22 diverse high-quality subsets—both existing and newly constructed—many of which derive from academic or professional sources.
View PDF on arXiv
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
View via Publisher
Multitask Prompted Training Enables Zero-Shot Task Generalization
TLDR
A system for easily mapping any natural language tasks into a human-readable prompted form and fine-tune a pretrained encoder-decoder model on this multitask mixture covering a wide variety of tasks.
View PDF on arXiv
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
TLDR
This work demonstrates on a variety of tasks how using CLIP to guide VQGAN produces higher visual quality outputs than prior, less flexible approaches like minDALL-E, GLIDE and Open-Edit, despite not being trained for the tasks presented.
View PDF on arXiv
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
TLDR
This work manually audit the quality of 205 language-specific corpora released with five major public datasets and recommends techniques to evaluate and improve multilingual corpora and discusses potential risks that come with low-quality data releases.
View on ACL
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license.…
View on ACL
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
TLDR
Evaluation of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters finds that model performance and calibration both improve with scale, but are poor in absolute terms.
View PDF on arXiv
Pitfalls in Machine Learning Research: Reexamining the Development Cycle
TLDR
This work follows the machine learning process from algorithm design to data collection to model evaluation, drawing attention to common pitfalls and providing practical recommendations for improvements.
View PDF on arXiv
Neural Language Models are Effective Plagiarists
TLDR
It is found that a student using GPT-J can complete introductory level programming assignments without triggering suspicion from MOSS, a widely used plagiarism detection tool.
View PDF on arXiv
Cut the CARP: Fishing for zero-shot story evaluation
TLDR
A strong correlation between human evaluation of stories and those of carp is shown, and model outputs more significantly correlate with corresponding human input than those language-model based methods which utilize finetuning or prompt engineering approaches.
View PDF on arXiv
...
...
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy, Terms of Service, and Dataset License