Share This Author
On the Opportunities and Risks of Foundation Models
This report provides a thorough account of the opportunities and risks of foundation models, ranging from their capabilities, to their applications, and what they are even capable of due to their emergent properties.
The EOS Decision and Length Extrapolation
- Benjamin Newman, John Hewitt, Percy Liang, Christopher D. Manning
- Computer ScienceBLACKBOXNLP
- 14 October 2020
It is found that -EOS substantially outperforms +EOS, for example extrapolating well to lengths 10 times longer than those seen at training time in a bracket closing task, as well as achieving a 40% improvement over +E OS in the difficult SCAN dataset length generalization task.
Refining Targeted Syntactic Evaluation of Language Models
It is found that TSE overestimates systematicity of language models, but that models score up to 40% better on verbs that they predict are likely in context and propose new metrics to capture each goal separately.
Ensembles and Cocktails: Robust Finetuning for Natural Language Generation
This work presents methods to combine the beneﬁts of full and lightweight ﬁnetuning, achieving strong performance both ID and OOD, and provides some explanatory theory in a multiclass logistic regression setting with a large number of classes.
P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts
What makes a P-Adapter successful is investigated and it is concluded that access to the LLM’s embeddings of the original natural language prompt, particularly the subject of the entity pair being asked about, is a significant factor.
Communication-based Evaluation for Natural Language Generation
This work argues instead for communication-based evaluations: assuming the purpose of an NLG system is to convey information to a reader/listener, it can directly evaluate its effectiveness at this task using the Rational Speech Acts model of pragmatic language use.
English-Chinese Name Machine Transliteration Using Search and Neural Network Models
It is found that search-based methods outperform deep learning ones, likely due to the relatively small number of English names with standard Chinese translations in the accessible dataset, and that incorporating syllable length heuristics and phonetic information into the search improves performance significantly.
Synth-tax: Modification of BERT Syntax Representations via Structural Probes
Large, pretrained, ‘black-box’ language models like BERT have recently risen to prominence, and we seek to understand how they represent linguistic information by causally relating models’…
Unsupervised Recovery of Tree Metrics from Learned Language Representations
- Benjamin Newman
- Computer Science
This work designs an unsupervised structural probe to using a variant of the EM algorithm and fails, but its failures point to future directions to further develop the idea of an un supervised probe.
Displaying Asynchronous Reactions to a Document: Two Goals and a Design
- Todd R. Davies, Benjamin Newman, Brendan T. O'Connor, Aaron Tam, L. Perry
- Computer ScienceArXiv
- 1 November 2006
The design of the new version of Deme, a Web-based platform for online deliberation, is described and it is argued that it achieves the two goals better than other recent designs.