• Corpus ID: 235670135

Learning Task Informed Abstractions

@article{Fu2021LearningTI,
  title={Learning Task Informed Abstractions},
  author={Xiang Fu and Ge Yang and Pulkit Agrawal and T. Jaakkola},
  journal={ArXiv},
  year={2021},
  volume={abs/2106.15612}
}
Current model-based reinforcement learning methods struggle when operating from complex visual scenes due to their inability to prioritize task-relevant features. To mitigate this problem, we propose learning Task Informed Abstractions (TIA) that explicitly separates rewardcorrelated visual features from distractors. For learning TIA, we introduce the formalism of Task Informed MDP (TiMDP) that is realized by training two models that learn visual features via cooperative reconstruction, but one… 

Task-Induced Representation Learning

It is found that representation learning generally improves sample efficiency on unseen tasks even in visually complex scenes and that task-induced representations can double learning ef-ciency compared to unsupervised alternatives.

INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL

The key principle behind the design is to integrate a term inspired by variational empowerment into a state-space model based on mutual information that prioritizes information that is correlated with action, thus ensur-ing that functionally relevant factors are captured during the RL process.

Robust Deep Reinforcement Learning via Multi-View Information Bottleneck

The results show that KL balancing can improve training of RSSM with a contrastive-learning-based or mutual-information-maximization objective, and the approach outperforms well-established baselines for generalization to unseen environments on the Procgen benchmark.

Look where you look! Saliency-guided Q-networks for visual RL tasks

SGQN vastly improves the generalization capability of Soft Actor-Critic agents and outperforms existing state-of-the-art methods on the Deepmind Control Generalization benchmark, setting a new reference in terms of training efficiency, generalization gap, and policy interpretability.

Denoised MDPs: Learning World Models Better Than the World Itself

This work categorizes information out in the wild into four types based on controllability and relation with reward, and formulate useful information as that which is both controllable and reward-relevant.

Task-Independent Causal State Abstraction

This paper introduces a novel state abstraction called Task-Independent Causal State Abstraction (TICSA), and observes that both the dynamics model and policies learned by the proposed method generalize well to unseen states and that TICSA also improves sample efficiency compared to learning without state abstraction.

Learning Representations for Pixel-based Control: What Matters and Why?

This paper presents a simple baseline approach that can learn meaningful representations with no metric-based learning, no data augmentations, no world-model learning, and no contrastive learning and hopes this view can motivate researchers to rethink representation learning when investigating how to best apply RL to real-world tasks.

BLAST: Latent Dynamics Models from Bootstrapping

The combination of adding a stop-gradient to the posterior, using a powerful auto-regressive model for the prior, and using a slowly updating target encoder, which the authors call BLAST, allows the world model to learn from signals present in both the reward and observations, improving efficiency on the authors' tested environment as well as being signi ficantly more robust to visual distractors.

Causal Dynamics Learning for Task-Independent State Abstraction

This paper introduces Causal Dynamics Learning for Task-Independent State Abstraction ( CDL), which first learns a theoretically proved causal dynamics model that removes unnecessary dependencies between state variables and the action, thus generalizing well to unseen states.

CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning

CARL is proposed, a collection of well-known RL environments extended to contextual RL problems to study generalization and allows first evidence that disentangling representation learning of the states from the policy learning with the context facilitates better generalization.

References

SHOWING 1-10 OF 66 REFERENCES

Learning Invariant Representations for Reinforcement Learning without Reconstruction

This work studies how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction, and proposes a method to learn robust latent representations which encode only the task-relevant information from observations.

Imagination-Augmented Agents for Deep Reinforcement Learning

Imagination-Augmented Agents (I2As), a novel architecture for deep reinforcement learning combining model-free and model-based aspects, shows improved data efficiency, performance, and robustness to model misspecification compared to several baselines.

Decoupling Representation Learning from Reinforcement Learning

A new unsupervised learning task, called Augmented Temporal Contrast (ATC), which trains a convolutional encoder to associate pairs of observations separated by a short time difference, under image augmentations and using a contrastive loss.

DARLA: Improving Zero-Shot Transfer in Reinforcement Learning

A new multi-stage RL agent, DARLA (DisentAngled Representation Learning Agent), which learns to see before learning to act, which significantly outperforms conventional baselines in zero-shot domain adaptation scenarios.

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

A simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks and demonstrating robustness to observational noise, surpassing existing approaches in this setting.

CURL: Contrastive Unsupervised Representations for Reinforcement Learning

CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted features and is the first image-based algorithm to nearly match the sample-efficiency of methods that use state-based features.

Deep Reinforcement and InfoMax Learning

An objective based on Deep InfoMax (DIM) is introduced which trains the agent to predict the future by maximizing the mutual information between its internal representation of successive timesteps.

Learning Latent Dynamics for Planning from Pixels

The Deep Planning Network (PlaNet) is proposed, a purely model-based agent that learns the environment dynamics from images and chooses actions through fast online planning in latent space using a latent dynamics model with both deterministic and stochastic transition components.

Deep visual foresight for planning robot motion

  • Chelsea FinnS. Levine
  • Computer Science
    2017 IEEE International Conference on Robotics and Automation (ICRA)
  • 2017
This work develops a method for combining deep action-conditioned video prediction models with model-predictive control that uses entirely unlabeled training data and enables a real robot to perform nonprehensile manipulation — pushing objects — and can handle novel objects not seen during training.

Model-Based Reinforcement Learning for Atari

Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models, is described and a comparison of several model architectures is presented, including a novel architecture that yields the best results in the authors' setting.
...