• Corpus ID: 219687812

How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds

@article{Ammanabrolu2020HowTA,
  title={How to Avoid Being Eaten by a Grue: Structured Exploration Strategies for Textual Worlds},
  author={Prithviraj Ammanabrolu and Ethan Tien and Matthew J. Hausknecht and Mark O. Riedl},
  journal={ArXiv},
  year={2020},
  volume={abs/2006.07409}
}
Text-based games are long puzzles or quests, characterized by a sequence of sparse and potentially deceptive rewards. They provide an ideal platform to develop agents that perceive and act upon the world using a combinatorially sized natural language state-action space. Standard Reinforcement Learning agents are poorly equipped to effectively explore such spaces and often struggle to overcome bottlenecks---states that agents are unable to pass through simply because they do not see the right… 
[PDF] Semantic Reader
31 Citations
Highly Influential Citations
6
Background Citations
15
Methods Citations
15
Results Citations
1

Figures and Tables from this paper

31 Citations

How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds

A reinforcement learning system is introduced that incorporates large-scale language modeling-based and commonsense reasoning-based pre-training to imbue the agent with relevant priors and leverages a factorized action space of action commands and dialogue, balancing between the two.

LANGUAGE ACTION VALUE ESTIMATES

This paper introduces Monte-Carlo planning with Language Action Value Estimates (MC-LAVE) that combines Monte- carlo tree search with language-driven exploration and presents a reinforcement learning approach built on MC-LAve, which alternates between MC- LAVE planning and supervised learning of the selfgenerated language actions.

Learning Knowledge Graph-based World Models of Textual Environments

This work focuses on the task of building world models of text-based game environments and frames this task as a Set of Sequences generation problem by exploiting the inherent structure of knowledge graphs and actions and introduces both a transformer-based multi-task architecture and a loss function to train it.

Modeling Worlds in Text

A dataset that enables the creation of learning agents that can build knowledge graph-based world models of interactive narratives and baseline models using rules-based, question-answering, and sequence learning approaches are provided in addition to an analysis of the data and corresponding learning tasks.

Story Shaping: Teaching Agents Human-like Behavior with Stories

This work introduces a technique, Story Shaping, in which a reinforcement learning agent infers tacit knowledge from an exemplar story of how to accomplish a task and intrinsically rewards itself for performing actions that make its current environment adhere to that of the inferred story world.

Situated Dialogue Learning through Procedural Environment Generation

An ablation study shows that this method of learning from the tail of a distribution results in significantly higher generalization abilities as measured by zero-shot performance on never-before-seen quests.

Playing Text-Based Games with Common Sense

It is concluded that agents that augment their beliefs about the world state with commonsense inferences are more robust to observational errors and omissions of common elements from text descriptions.

Potential-based reward shaping for learning to play text-based adventure games

This paper adapts the soft-actor-critic (SAC) algorithm to the text-based environment, and considers a dynamically learned value function as a potential function for shaping the learner's original sparse reward signals.

Case-based Reasoning for Better Generalization in Text-Adventure Games

A general method inspired by case-based reasoning to train agents and generalize out of the training distribution, which consistently improves existing methods, obtains good out-of-distribution generalization, and achieves new state- of-the-art results on widely used environments.

An Analysis of Deep Reinforcement Learning Agents for Text-based Games

A new RL agent is proposed that e-ectively learns from limited game information, by aligning action text with key words in observation text by using an action-state attention module in the model.

References

SHOWING 1-10 OF 32 REFERENCES

Graph Constrained Reinforcement Learning for Natural Language Action Spaces

KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space is presented, arguing that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions.

Algorithmic Improvements for Deep Reinforcement Learning applied to Interactive Fiction

This paper proposes a contextualisation mechanism, based on accumulated reward, which simplifies the learning problem and mitigates partial observability, and studies different methods that rely on the notion that most actions are ineffectual in any given situation.

Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning

A deep reinforcement learning architecture that represents the game state as a knowledge graph which is learned during exploration that is used to prune the action space, enabling more efficient exploration.

Exploration Based Language Learning for Text-Based Games

This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games and shows that the learned policy can generalize better than existing solutions to unseen games without using any restriction on the action space.

Learning Dynamic Knowledge Graphs to Generalize on Text-Based Games

This work introduces a novel transformer-based sequence-to-sequence model that constructs a “belief” KG from raw text observations of the environment, dynamically updating this belief graph at every game step as it receives new observations.

LeDeepChef: Deep Reinforcement Learning Agent for Families of Text-Based Games

This work presents a deep RL agent—LeDeepChef—that shows generalization capabilities to never-before-seen games of the same family with different environments and task descriptions, and uses an actor-critic framework and prune the action-space to build an agent that achieves high scores across a whole family of games.

Go-Explore: a New Approach for Hard-Exploration Problems

A new algorithm called Go-Explore, which exploits the following principles to remember previously visited states, solve simulated environments through any available means, and robustify via imitation learning, which results in a dramatic performance improvement on hard-exploration problems.

Counting to Explore and Generalize in Text-based Games

A recurrent RL agent with an episodic exploration mechanism that helps discovering good policies in text-based game environments and observes that the agent learns policies that generalize to unseen games of greater difficulty.

What Can You Do with a Rock? Affordance Extraction via Word Embeddings

This paper applies a method for affordance extraction via word embeddings trained on a Wikipedia corpus to a reinforcement learning agent in a text-only environment and shows that affordance-based action selection improves performance most of the time.

Learning Options in Reinforcement Learning

This paper empirically explores a simple approach to creating options based on the intuition that states that are frequently visited on system trajectories, could prove to be useful subgoals, and proposes a greedy algorithm for identifying subgoal counts based on state visitation counts.