• Corpus ID: 211020594

On the interaction between supervision and self-play in emergent communication

@article{Lowe2020OnTI,
  title={On the interaction between supervision and self-play in emergent communication},
  author={Ryan Lowe and Abhinav Gupta and Jakob N. Foerster and Douwe Kiela and Joelle Pineau},
  journal={ArXiv},
  year={2020},
  volume={abs/2002.01093}
}
A promising approach for teaching artificial agents to use natural language involves using human-in-the-loop training. However, recent work suggests that current machine learning methods are too data inefficient to be trained in this way from scratch. In this paper, we investigate the relationship between two categories of learning signals with the ultimate goal of improving sample efficiency: imitating human language data via supervised learning, and maximizing reward in a simulated multi… 

Figures and Tables from this paper

Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning

A new way for combining the two types of learning based on the idea of reranking language model samples is introduced, and it is shown that this method outperforms others in communicating with humans in a visual referential communication task.

Self-play for Data Efficient Language Acquisition

It is shown that using self-play as a substitute for direct supervision enables the agent to transfer its knowledge across roles and make better inferences about the ground truth lexicon using only a handful of interactions with the oracle.

SocialAI 0.1: Towards a Benchmark to Stimulate Research on Socio-Cognitive Abilities in Deep Reinforcement Learning Agents

This work explains how concepts from cognitive sciences could help AI to draw a roadmap towards human-like intelligence, with a focus on its social dimensions and studies the limits of a recent SOTA Deep RL approach when tested on a first grid-world environment from the upcoming SocialAI, a benchmark to assess the social skills of Deep RL agents.

Analyzing structural priors in multi-agent communication

This work explores various structural priors in multi-agent communication and proposes a novel graph referential game that empirically shows that graph neural networks induce a better compositional language prior and a stronger generalization to out-of-domain data.

Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations

This work shows that under realistic assumptions, a non-uniform distribution of intents and a common-knowledge energy cost, agents that learn to communicate via actuating their joints in a 3D environment can find protocols that generalize to novel partners.

Exploring Structural Inductive Biases in Emergent Communication

This work explores various structural priors in multi-agent communication and proposes a novel graph referential game that empirically shows that graph neural networks induce a better compositional language prior and a stronger generalization to out-of-domain data.

Emergent Multi-Agent Communication in the Deep Learning Era

This article surveys representative recent language emergence studies from a scientific and applied perspective about how language evolves in communities of deep agents and its emergent features are understood.

Supervised Seeded Iterated Learning for Interactive Language Learning

This paper introduces Supervised Seeded Iterated Learning to combine both methods to minimize their respective weaknesses, and shows the effectiveness of \algo in the language-drift translation game.

The emergence of visual semantics through communication games

This work investigates the effect of the feature extractor’s weights and of the task being solved on the visual semantics learned by the models, and demonstrates that communication systems which capture visual semantics can be learned in a completely self-supervised manner by playing the right types of game.

Emergent Communication Pretraining for Few-Shot Machine Translation

It is shown that grounding communication on images—as a crude approximation of real-world environments—inductively biases the model towards learning natural languages, and the potential of emergent communication pretraining for both natural language processing tasks in resource-poor settings and extrinsic evaluation of artificial languages is revealed.

References

SHOWING 1-10 OF 51 REFERENCES

Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols

This work studies a setting where two agents engage in playing a referential game and develops a communication protocol necessary to succeed in this game, requiring that messages they exchange are in the form of a language.

Multi-Agent Cooperation and the Emergence of (Natural) Language

It is shown that two networks with simple configurations are able to learn to coordinate in the referential game and how to make changes to the game environment to cause the "word meanings" induced in the game to better reflect intuitive semantic properties of the images.

Natural Language Does Not Emerge ‘Naturally’ in Multi-Agent Dialog

This paper presents a sequence of ‘negative’ results culminating in a ‘positive’ one – showing that while most agent-invented languages are effective, they are decidedly not interpretable or compositional.

Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning

A new way for combining the two types of learning based on the idea of reranking language model samples is introduced, and it is shown that this method outperforms others in communicating with humans in a visual referential communication task.

Learning Existing Social Conventions via Observationally Augmented Self-Play

It is observed that augmenting MARL with a small amount of imitation learning greatly increases the probability that the strategy found by MARL fits well with the existing social convention, even in an environment where standard training methods very rarely find the true convention of the agent's partners.

Mastering emergent language: learning to guide in simulated navigation

An autonomous agent is introduced that uses discrete communication to interactively guide other agents to navigate and act on a simulated environment and how new agents become less dependent on this guidance as training progresses.

Emergent Linguistic Phenomena in Multi-Agent Communication Games

It is concluded that intricate properties of language evolution need not depend on complex evolved linguistic capabilities, but can emerge from simple social exchanges between perceptually-enabled agents playing communication games.

Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

Empirical results demonstrate that influence leads to enhanced coordination and communication in challenging social dilemma environments, dramatically increasing the learning curves of the deep RL agents, and leading to more meaningful learned communication protocols.

Learning to Communicate with Deep Multi-Agent Reinforcement Learning

By embracing deep neural networks, this work is able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability.

Emergence of Compositional Language with Deep Generational Transmission

It is shown that this implicit cultural transmission encourages the resulting languages to exhibit better compositional generalization and suggest how elements of cultural dynamics can be further integrated into populations of deep agents.
...