Masked Autoencoding for Scalable and Generalizable Decision Making

@article{Liu2022MaskedAF,
  title={Masked Autoencoding for Scalable and Generalizable Decision Making},
  author={Fangchen Liu and Hao Liu and Aditya Grover and P. Abbeel},
  journal={ArXiv},
  year={2022},
  volume={abs/2211.12740}
}
We are interested in learning scalable agents for reinforcement learning that can learn from large-scale, diverse sequential data similar to current large vision and language models. To this end, this paper presents masked decision prediction (MaskDP), a simple and scalable self-supervised pretraining method for reinforcement learning (RL) and behavioral cloning (BC). In our MaskDP approach, we employ a masked autoencoder (MAE) to state-action trajectories, wherein we randomly mask state and… 
[PDF] Semantic Reader
3 Citations
Highly Influential Citations
1
Methods Citations
2

3 Citations

UniMASK: Unified Inference in Sequential Decision Problems

The Uni [MASK] framework is introduced, which provides a unified way to specify models which can be trained on many different sequential decision making tasks, and it is shown that a single Uni [ MASK] model is often capable of carrying out many tasks with performance similar to or better than single-task models.

On Transforming Reinforcement Learning by Transformer: The Development Trajectory

This survey collects and dissects recent advances on transforming RL by transformer (transformer-based RL or TRL), in order to explore its development trajectory and future trend, and examines the main applications of TRL in robotic manipulation, text-based games, navigation and autonomous driving.

Aerial Image Object Detection With Vision Transformer Detector (ViTDet)

The empirical study shows that ViTDet’s simple design achieves good performance on natural scene images and can be easily embedded into any detector architecture and that it achieves the competitive performance for oriented bounding box (OBB) object detection.

References

SHOWING 1-10 OF 37 REFERENCES

UniMASK: Unified Inference in Sequential Decision Problems

The Uni [MASK] framework is introduced, which provides a unified way to specify models which can be trained on many different sequential decision making tasks, and it is shown that a single Uni [ MASK] model is often capable of carrying out many tasks with performance similar to or better than single-task models.

Masked World Models for Visual Control

A visual model-based RL framework that decouples visual representation learning and dynamics learning is introduced that achieves state-of-the-art performance on a variety of visual robotic tasks from Meta-world and RLBench.

Data-Efficient Reinforcement Learning with Self-Predictive Representations

The method, Self-Predictive Representations (SPR), trains an agent to predict its own latent state representations multiple steps into the future using an encoder which is an exponential moving average of the agent’s parameters and a learned transition model.

Pretraining Representations for Data-Efficient Reinforcement Learning

This work uses unlabeled data to pretrain an encoder which is then finetuned on a small amount of task-specific data, and employs a combination of latent dynamics modelling and unsupervised goal-conditioned RL to encourage learning representations which capture diverse aspects of the underlying MDP.

Reinforcement Learning with Action-Free Pre-Training from Videos

A framework that learns representations useful for understanding the dynamics via generative pretraining on videos that improves both performances and sample-efficiency of vision-based RL in a variety of manipulation and locomotion tasks is introduced.

Decoupling Representation Learning from Reinforcement Learning

A new unsupervised learning task, called Augmented Temporal Contrast (ATC), which trains a convolutional encoder to associate pairs of observations separated by a short time difference, under image augmentations and using a contrastive loss.

Decision Transformer: Reinforcement Learning via Sequence Modeling

Despite its simplicity, Decision Transformer matches or exceeds the performance of state-of-the-art model-free offline RL baselines on Atari, OpenAI Gym, and Key-to-Door tasks.

URLB: Unsupervised Reinforcement Learning Benchmark

The Unsupervised Reinforcement Learning Benchmark (URLB) is introduced, providing twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods and finds that the implemented baselines make progress but are not able to solve URLB.

Reinforcement Learning with Prototypical Representations

Proto-RL is a self-supervised framework that ties representation learning with exploration through prototypical representations that serve as a summarization of the exploratory experience of an agent as well as a basis for representing observations.

Reinforcement Learning as One Big Sequence Modeling Problem

This work explores how RL can be reframed as “one big sequence modeling” problem, using state-of-the-art Transformer architectures to model distributions over sequences of states, actions, and rewards.