Measuring and Improving Consistency in Pretrained Language Models
- Yanai Elazar, Nora Kassner, Yoav Goldberg
- Computer ScienceTransactions of the Association for Computational…
- 2 February 2021
The creation of PARAREL, a high-quality resource of cloze-style query English paraphrases, and analysis of the representational spaces of PLMs suggest that they have a poor structure and are currently not suitable for representing knowledge in a robust way.
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
Two new probing tasks analyzing factual knowledge stored in Pretrained Language Models are proposed and it is found that PLMs do not distinguish between negated and non-negated cloze questions, and PLMs are easily distracted by misprimes.
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models
- Nora Kassner, Philipp Dufter, Hinrich Schütze
- Computer Science, LinguisticsConference of the European Chapter of the…
- 1 February 2021
This work translates the established benchmarks TREx and GoogleRE into 53 languages and finds that using mBERT as a knowledge base yields varying performance across languages and pooling predictions across languages improves performance.
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief
- Nora Kassner, Oyvind Tafjord, Hinrich Schutze, Peter Clark
- Computer ScienceConference on Empirical Methods in Natural…
- 29 September 2021
This work describes two mechanisms to improve belief consistency in the overall system, enabling PTLM-based architectures with a systematic notion of belief to construct a more coherent picture of the world, and improve over time without model retraining.
Are Pretrained Language Models Symbolic Reasoners over Knowledge?
- Nora Kassner, Benno Krojer, Hinrich Schütze
- Computer ScienceConference on Computational Natural Language…
- 18 June 2020
This is the first study that investigates the causal relation between facts present in training and facts learned by the PLM, and shows that PLMs seem to learn to apply some symbolic reasoning rules correctly but struggle with others, including two-hop reasoning.
Anisotropic cerebral vascular architecture causes orientation dependency in cerebral blood flow and volume measured with dynamic susceptibility contrast magnetic resonance imaging
- E. Hernández-Torres, Nora Kassner, Alexander Rauscher
- MedicineJournal of cerebral blood flow and metabolism…
- 21 July 2016
Numerical simulations agreed with the measured data, showing that one-third of the white matter vascular volume is comprised of vessels running in parallel with the fibre tracts.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
BLOOM is a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers and achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning.
Negated LAMA: Birds cannot fly
It is found that pretrained language models are equally prone to generate facts ("birds can fly") and their negation ("birds cannot fly").
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA
BERT-kNN outperforms BERT on cloze-style QA by large margins without any further training and excels for rare facts.
Static Embeddings as Efficient Knowledge Bases?
- Philipp Dufter, Nora Kassner, Hinrich Schütze
- Computer ScienceNorth American Chapter of the Association for…
- 14 April 2021
It is shown that, when restricting the output space to a candidate set, simple nearest neighbor matching using static embeddings performs better than PLMs, while just using 0.3% of energy for training.