• Corpus ID: 232045915

On The Effect of Auxiliary Tasks on Representation Dynamics

@inproceedings{Lyle2021OnTE,
  title={On The Effect of Auxiliary Tasks on Representation Dynamics},
  author={Clare Lyle and Mark Rowland and Georg Ostrovski and Will Dabney},
  booktitle={AISTATS},
  year={2021}
}
While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the… 
What makes useful auxiliary tasks in reinforcement learning: investigating the effect of the target policy
Auxiliary tasks have been argued to be useful for representation learning in reinforcement learning. Although many auxiliary tasks have been empirically shown to be effective for accelerating
Transfer RL across Observation Feature Spaces via Model-Based Regularization
TLDR
A novel algorithm is proposed which extracts the latent-space dynamics in the source task, and transfers the dynamics model to the target task to use as a model-based regularizer, and significantly improves the efficiency and stability of learning in thetarget task.
Learning Dynamics and Generalization in Reinforcement Learning
TLDR
This paper analyzes the learning dynamics of temporal difference algorithms to gain novel insight into the tension between these two objectives of reinforcement learning, and shows theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training.
Learning State Representations from Random Deep Action-conditional Predictions
TLDR
This work shows that deep action-conditional TD networks with random structures that create random prediction-questions about random features yield state representations that are competitive with state-of-the-art hand-crafted value prediction and pixel control auxiliary tasks in both Atari games and DeepMind Lab tasks.
Understanding and Preventing Capacity Loss in Reinforcement Learning
TLDR
It is concluded that preventing capacity loss is crucial to enable agents to maximally benefit from the learning signals they obtain throughout the entire training trajectory.
On the Generalization of Representations in Reinforcement Learning
TLDR
This work provides an in-formative bound on the generalization error arising from a specific state representation based on the notion of e-ective dimension which measures the degree to which knowing thevalue at one state informs the value at other states.
Representation Gap in Deep Reinforcement Learning
TLDR
This work proposes a simple but effective framework Policy Optimization from Preventing Representation Overlaps (POPRO), which regularizes the policy evaluation phase through differing the representation of action value function from its target and provides the convergence rate guarantee of POPRO.
Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
TLDR
This paper investigates causes of instability when using data augmentation in common off-policy RL algorithms and proposes a simple yet effective technique for stabilizing this class of algorithms under augmentation, and achieves generalization results competitive with state-of-the-art methods for image-based RL in environments with unseen visuals.
Multi-Game Decision Transformers
TLDR
It is shown that a single transformer-based model – with a single set of weights – trained purely offline can play a suite of up to 46 Atari games simultaneously at close-to-human performance.
Uncertainty-aware Low-Rank Q-Matrix Estimation for Deep Reinforcement Learning
TLDR
It is reported that decreasing rank of Q-matrix widely exists during learning process across a series of continuous control tasks for different popular algorithms, and a positive correlation between value matrix rank and value estimation uncertainty is revealed.
...
...

References

SHOWING 1-10 OF 72 REFERENCES
Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning
Learning a good representation is an essential component for deep reinforcement learning (RL). Representation learning is especially important in multitask and partially observable settings where
Eigenoption Discovery through the Deep Successor Representation
TLDR
This paper proposes an algorithm that discovers eigenoptions while learning non-linear state representations from raw pixels, and exploits recent successes in the deep reinforcement learning literature and the equivalence between proto-value functions and the successor representation.
Reinforcement Learning with Unsupervised Auxiliary Tasks
TLDR
This paper significantly outperforms the previous state-of-the-art on Atari, averaging 880\% expert human performance, and a challenging suite of first-person, three-dimensional \emph{Labyrinth} tasks leading to a mean speedup in learning of 10$\times$ and averaging 87\% Expert human performance on Labyrinth.
Terminal Prediction as an Auxiliary Task for Deep Reinforcement Learning
TLDR
This paper contributes a novel self-supervised auxiliary task, i.e., Terminal Prediction (TP), estimating temporal closeness to terminal states for episodic tasks, to help representation learning by letting the agent predict how close it is to a terminal state, while learning its control policy.
Adaptive Auxiliary Task Weighting for Reinforcement Learning
TLDR
This work proposes a principled online learning algorithm that dynamically combines different auxiliary tasks to speed up training for reinforcement learning and achieves significant speedup compared to previous heuristic approches of adapting auxiliary task weights.
Learning by Playing - Solving Sparse Reward Tasks from Scratch
TLDR
The key idea behind the method is that active (learned) scheduling and execution of auxiliary policies allows the agent to efficiently explore its environment - enabling it to excel at sparse reward RL.
Decoupling Representation Learning from Reinforcement Learning
TLDR
A new unsupervised learning task, called Augmented Temporal Contrast (ATC), which trains a convolutional encoder to associate pairs of observations separated by a short time difference, under image augmentations and using a contrastive loss.
Adapting Auxiliary Losses Using Gradient Similarity
TLDR
This work proposes to use the cosine similarity between gradients of tasks as an adaptive weight to detect when an auxiliary loss is helpful to the main loss and shows that the approach is guaranteed to converge to critical points of the main task.
A Geometric Perspective on Optimal Representations for Reinforcement Learning
TLDR
A new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions is proposed, and it is demonstrated that using value functions as auxiliary tasks corresponds to an expected-error relaxation of the formulation.
DeepMDP: Learning Continuous Latent Space Models for Representation Learning
TLDR
This work introduces the concept of a DeepMDP, a parameterized latent space model that is trained via the minimization of two tractable losses: prediction of rewards and prediction of the distribution over next latent states, and shows that the optimization of these objectives guarantees the quality of the latent space as a representation of the state space.
...
...