Skip to search formSkip to main contentSkip to account menu

On the Opportunities and Risks of Foundation Models

This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities, to their applications, and what they are even capable of due to their emergent properties.
View PDF on arXiv (opens in a new tab)

Diffusion-LM Improves Controllable Text Generation

A new non-autoregressive language model based on continuous diffusions that iteratively denoises a sequence of Gaussian vectors into word vectors, yielding a sequences of intermediate latent variables that enables a simple gradient-based algorithm to perform complex, controllable generation tasks.
View PDF on arXiv (opens in a new tab)

Emergent Abilities of Large Language Models

Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we…
View PDF on arXiv (opens in a new tab)

Holistic Evaluation of Language Models

Holistic Evaluation of Language Models (HELM) is presented to improve the transparency of language models and intends for HELM to be a living benchmark for the community, continuously updated with new scenarios, metrics, and models.
View PDF on arXiv (opens in a new tab)

Jury Learning: Integrating Dissenting Voices into Machine Learning Models

A deep learning architecture that models every annotator in a dataset, samples from annotators’ models to populate the jury, then runs inference to classify enables juries that dynamically adapt their composition, explore counterfactuals, and visualize dissent.
View on ACM (opens in a new tab)

Contrastive Decoding: Open-ended Text Generation as Optimization

Contrastive decoding (CD), a more reliable search objective that returns the difference between likelihood under a large LM and a small LM, is proposed, which requires zero training, and produces higher quality text than decoding from the larger LM alone.
View PDF on arXiv (opens in a new tab)

Is Importance Weighting Incompatible with Interpolating Classifiers?

Surprisingly, the theory shows that using weights that are obtained by exponentiating the classical unbiased importance weights can improve performance, and the loss function can outperform reweighted cross-entropy by as much as 9% in test accuracy.
View PDF on arXiv (opens in a new tab)

Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning

This work studies this question through a carefully controlled comparison of two approaches in terms of their ability to learn representations that generalize to downstream classification tasks, finding that when the pre-training dataset meets certain criteria—it is suf ficiently large and contains descriptive captions with low variability—image-only methods do not match CLIP’s transfer performance, even when they are trained with more image data.
View PDF on arXiv (opens in a new tab)

Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale

This work investigates image-generation models and finds that they amplify dangerous and complex stereotypes, which are difficult to predict and not easily mitigated by users or model owners.
View PDF on arXiv (opens in a new tab)

Language modeling via stochastic processes

Compared to domain-specific methods and fine-tuning GPT2 across a variety of text domains, TC improves performance on text infilling and discourse coherence and preserves the text structure on long text generation settings.
View PDF on arXiv (opens in a new tab)
By clicking accept or continuing to use the site, you agree to the terms outlined in our Privacy Policy (opens in a new tab), Terms of Service (opens in a new tab), and Dataset License (opens in a new tab)