Language (Re)modelling: Towards Embodied Language Understanding

  title={Language (Re)modelling: Towards Embodied Language Understanding},
  author={Ronen Tamari and Cheng Shani and Tom Hope and Miriam R. L. Petruck and Omri Abend and Dafna Shahaf},
While natural language understanding (NLU) is advancing rapidly, today’s technology differs from human-like language understanding in fundamental ways, notably in its inferior efficiency, interpretability, and generalization. This work proposes an approach to representation and learning based on the tenets of embodied cognitive linguistics (ECL). According to ECL, natural language is inherently executable (like programming languages), driven by mental simulation and metaphoric mappings over… 

Figures and Tables from this paper

Corpus-based Metaphor Analysis through Graph Theoretical Methods

As a contribution to metaphor analysis, we introduce a statistical, data-based investigation with empirical analysis of long-standing conjectures and a first-ever empirical exploration of the

Contextualized Sensorimotor Norms: multi-dimensional measures of sensorimotor strength for ambiguous English words, in context

Most large language models are trained on 001 linguistic input alone, yet humans appear to 002 ground their understanding of words in senso- 003 rimotor experience. A natural solution is to 004

Breakpoint Transformers for Modeling and Tracking Intermediate Beliefs

The feasibility of incorporating the main breakpoint transformer, based on T5, into more complex reasoning pipelines, is obtained and SOTA performance on the three-tiered reasoning challenge for the TRIP benchmark is obtained.

Towards Socially Intelligent Agents with Mental State Transition and Human Value

A hybrid mental state parser that extracts information from both the dialogue and event observations and maintains a graphical representation of the agent’s mind and a transformer-based value model that learns human preferences from the human value dataset, ValueNet.

ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments

This work uses the AI2-THOR simulated environment to produce a controlled setup in which an agent has to determine what the correct after-image is among a set of possible candidates, and suggests that only models that have a very structured representation of the actions together with powerful visual features can perform well on the task.

The VoxWorld Platform for Multimodal Embodied Agents

We present a five-year retrospective on the development of the VoxWorld platform, first introduced as a multimodal platform for modeling motion language, that has evolved into a platform for rapidly

What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris

The power of word embeddings is attributed to the linguistic theory that similar words will appear in similar contexts. This idea is specifically invoked by noting that “you shall know a word by the

A Metaverse: Taxonomy, Components, Applications, and Open Challenges

This paper divides the concepts and essential techniques necessary for realizing the Metaverse into three components (i.e., hardware, software, and contents) and three approaches and describes essential methods based on three components and techniques to Metaverse’s representative Ready Player One, Roblox, and Facebook research in the domain of films, games, and studies.



Conceptual Alignment: How Brains Achieve Mutual Understanding

Ingredients of intelligence: From classic debates to an engineering roadmap

This response covers three main dimensions of disagreement: nature versus nurture, coherent theories versus theory fragments, and symbolic versus sub-symbolic representations in artificial intelligence and cognitive science.

On Making Reading Comprehension More Comprehensive

This work justifies a question answering approach to reading comprehension and describes the various kinds of questions one might use to more fully test a system’s comprehension of a passage, moving beyond questions that only probe local predicate-argument structures.

Question Answering is a Format; When is it Useful?

It is argued that question answering should be considered a format which is sometimes useful for studying particular phenomena, not a phenomenon or task in itself.

Learning to activate logic rules for textual reasoning

The Consciousness Prior

A new prior is proposed for learning representations of high-level concepts of the kind the authors manipulate with language, inspired by cognitive neuroscience theories of consciousness, that makes it natural to map conscious states to natural language utterances or to express classical AI knowledge in a form similar to facts and rules.

Simpler Context-Dependent Logical Forms via Model Projections

This work considers the task of learning a context-dependent mapping from utterances to denotations, and performs successive projections of the full model onto simpler models that operate over equivalence classes of logical forms.

The emulation theory of representation: Motor control, imagery, and perception

  • R. Grush
  • Psychology
    Behavioral and Brain Sciences
  • 2004
The emulation theory of representation is developed and explored as a framework that can revealingly synthesize a wide variety of representational functions of the brain, including reasoning, theory of mind phenomena, and language.

Question answering is a format

  • 2019

Ecological Semantics: Programming Environments for Situated Language Understanding

It is argued that models must begin to understand and program in the language of affordances both for online, situated discourse comprehension, as well as large-scale, offline common-sense knowledge mining, in an environment-oriented ecological semantics.