Hypernetworks in Meta-Reinforcement Learning

@inproceedings{Beck2022HypernetworksIM,
  title={Hypernetworks in Meta-Reinforcement Learning},
  author={Jacob Beck and Matthew Thomas Jackson and Risto Vuorio and Shimon Whiteson},
  booktitle={Conference on Robot Learning},
  year={2022}
}
Training a reinforcement learning (RL) agent on a real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim to improve sample efficiency by generalizing over a distribution of related tasks. However, doing so is difficult in practice: In multi-task RL, state of the art methods often fail to outperform a degenerate solution that simply learns each task separately. Hypernetworks are a promising path forward since they replicate the separate… 
[PDF] Semantic Reader
2 Citations
Background Citations
2
Methods Citations
1

Figures and Tables from this paper

2 Citations

A Survey of Meta-Reinforcement Learning

This survey describes the meta-RL problem setting in detail as well as its major variations and discusses howMeta-RL research can be clustered based on the presence of a task distribution and the learning budget available for each individual task.

Universal Morphology Control via Contextual Modulation

A hierarchical architecture to better model the dependency of a robot's control policy on its morphology context via contextual modulation is proposed, which includes two key submodules: Instead of enforcing hard parameter sharing across robots, hypernetworks are used to generate morphology-dependent control parameters and a morphology- dependent attention mechanism to modulate the interactions between different limbs in a robot.

35 References

Recomposing the Reinforcement Learning Building Blocks with Hypernetworks

This work suggests using a Hypernetwork architecture where a primary network determines the weights of a conditional dynamic network and shows that this approach improves the gradient approximation and reduces the learning step variance, which both accelerates learning and improves the final performance.

Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning

An open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks is proposed to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks.

Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

This paper develops an off-policy meta-RL algorithm that disentangles task inference and control and performs online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience.

Linear Representation Meta-Reinforcement Learning for Instant Adaptation

Experiments on standard continuous-control meta-RL benchmarks show FLAP presenting significantly stronger performance on out-of-distribution tasks with up to double the average return and up to 8X faster adaptation run-time speeds when compared to prior methods.

Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

This work defines a novel method of multitask and transfer learning that enables an autonomous agent to learn how to behave in multiple tasks simultaneously, and then generalize its knowledge to new domains, and uses Atari games as a testing environment to demonstrate these methods.

Meta reinforcement learning as task inference

This work proposes a method that separately learns the policy and the task belief by taking advantage of various kinds of privileged information, which can be very effective at solving standard meta-RL environments, as well as a complex continuous control environment with sparse rewards and requiring long-term memory.

Distral: Robust multitask reinforcement learning

This work proposes a new approach for joint training of multiple tasks, which it refers to as Distral (Distill & transfer learning), and shows that the proposed learning process is more robust and more stable---attributes that are critical in deep reinforcement learning.

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning…

Multi-Task Reinforcement Learning without Interference

This work develops a general approach that can change the multi-task optimization landscape to alleviate conflicting gradients across tasks and introduces two instantiations of this approach that prevent gradients for different tasks from interfering with one another.

Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation

This paper proposes a multimodal MAML (MMAML) framework, which is able to modulate its meta-learned prior parameters according to the identified mode, allowing more efficient fast adaptation and demonstrating the effectiveness of the model in modulating the meta-learning prior in response to the characteristics of tasks.