• Corpus ID: 225103287

A Visuospatial Dataset for Naturalistic Verb Learning

  title={A Visuospatial Dataset for Naturalistic Verb Learning},
  author={Dylan Ebert and Ellie Pavlick},
We introduce a new dataset for training and evaluating grounded language models. Our data is collected within a virtual reality environment and is designed to emulate the quality of language data to which a pre-verbal child is likely to have access: That is, naturalistic, spontaneous speech paired with richly grounded visuospatial context. We use the collected data to compare several distributional semantics models for verb learning. We evaluate neural models based on 2D (pixel) features as… 

Figures and Tables from this paper

Pretraining on Interactions for Learning Grounded Affordance Representations

A neural network is trained to predict objects? trajectories in a simulated interaction and it is shown that the network?s latent representations differentiate between both observed and unobserved affordances.

Do Trajectories Encode Verb Meaning?

Distributional models learn representations of words from text, but are criticized for their lack of grounding, or the linking of text to the non-linguistic world. Grounded language models have had

Cross-Situational Word Learning With Multimodal Neural Networks

Two variants of a multimodal neural network architecture are considered and shown that these networks can learn word-referent mappings from a single epoch of training, mimicking the amount of training commonly found in cross-situational word learning experiments.



Understanding Grounded Language Learning Agents

This work proposes a novel way to visualise and analyse semantic representation in grounded language learning agents that yields a plausible computational account of the observed effects and applies experimental paradigms from developmental psychology to this agent.

Understanding Early Word Learning in Situated Artificial Agents

This paper focuses on a simple neural network-based language learning agent, trained via policy-gradient methods, which can interpret single-word instructions in a simulated 3D world and proposes a novel method for visualising semantic representations in the agent.

A multimodal corpus for the evaluation of computational models for (grounded) language acquisition

A German multimodal corpus designed to support the development and evaluation of models learning rather complex grounded linguistic structures, e.g. syntactic patterns, from sub-symbolic input is described.

Grounded Models of Semantic Representation

Experimental results show that a closer correspondence to human data can be obtained by uncovering latent information shared among the textual and perceptual modalities rather than arriving at semantic knowledge by concatenating the two.

Human simulations of vocabulary learning

Neural Naturalist: Generating Fine-Grained Image Comparisons

A new model is proposed called Neural Naturalist that uses a joint image encoding and comparative module to generate comparative language, and the results indicate promising potential for neural models to explain differences in visual embedding space using natural language.

How Can We Accelerate Progress Towards Human-like Linguistic Generalization?

This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, and advocates for supplementing or replacing PAID with paradigms that reward architectures that generalize as quickly and robustly as humans.

Wordbank: an open repository for developmental vocabulary data*

Abstract The MacArthur-Bates Communicative Development Inventories (CDIs) are a widely used family of parent-report instruments for easy and inexpensive data-gathering about early language

Combining Language and Vision with a Multimodal Skip-gram Model

Since they propagate visual information to all words, the MMSKIP-GRAM models discover intriguing visual properties of abstract words, paving the way to realistic implementations of embodied theories of meaning.

Why Nouns Trump Verbs in Word Learning: New Evidence from Children and Adults in the Human Simulation Paradigm

The HSP task is modified to accommodate children and represents the first empirical demonstration that young children's noun advantage may be attributable, at least in part, to the distinct linguistic requirements underlying the acquisition of nouns and verbs.