Curriculum Learning with Hindsight Experience Replay for Sequential Object Manipulation Tasks

  title={Curriculum Learning with Hindsight Experience Replay for Sequential Object Manipulation Tasks},
  author={Binyamin Manela and Armin Biess},
  journal={Neural networks : the official journal of the International Neural Network Society},
  • Binyamin ManelaA. Biess
  • Published 21 August 2020
  • Computer Science
  • Neural networks : the official journal of the International Neural Network Society

Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewards

A novel self-guided continual RL framework, Relay-HER, which decomposes a sequential task into new sub-tasks with increasing complexity and ensures that the simplest sub-task can be learned quickly by utilizing Hindsight Experience Replay (HER).

Relay Hindsight Experience Replay: Continual Reinforcement Learning for Robot Manipulation Tasks with Sparse Rewards

A novel model-free continual RL algorithm, called Relay-HER (RHER), which decomposes and rearranges the original long-horizon task into new sub-tasks with incremental complexity, and proposes a Random-Mixed Exploration Strategy (RMES), in which the achieved goals of the sub-task with higher complexity are quickly changed under the guidance of the one with lower complexity.

An Open-Source Multi-Goal Reinforcement Learning Environment for Robotic Manipulation with Pybullet

This work re-implements the OpenAI Gym multi-goal robotic manipulation environment, originally based on the commercial Mujoco engine, onto the open-source Pybullet engine, and uses a simple, human-prior-based curriculum learning method to benchmark the multi-step manipulation tasks.

Real Robot Challenge Phase 2

  • Computer Science
  • 2021
Ultimately, Phase 2 of the Real Robot Challenge proves significantly more difficult than Phase 1 and the method struggles to solve the task, both in simulation or indeed reality.

Curriculum Learning: A Survey

This survey shows how limits have been tackled in the literature, and presents curriculum learning instantiations for various tasks in machine learning, and constructs a multi-perspective clustering algorithm, linking the discovered clusters with the taxonomy.



Autonomous Task Sequencing for Customized Curriculum Design in Reinforcement Learning

This paper forms the design of a curriculum as a Markov Decision Process, which directly models the accumulation of knowledge as an agent interacts with tasks, and proposes a method that approximates an execution of an optimal policy in this MDP to produce an agent-specific curriculum.

Hindsight Experience Replay

A novel technique is presented which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering and may be seen as a form of implicit curriculum.

Reverse Curriculum Generation for Reinforcement Learning

This work proposes a method to learn goal-oriented tasks without requiring any prior knowledge other than obtaining a single state in which the task is achieved, and generates a curriculum of start states that adapts to the agent's performance, leading to efficient training on goal- oriented tasks.

Learning Curriculum Policies for Reinforcement Learning

The method is extended to handle multiple transfer learning algorithms, and it is shown for the first time that a curriculum policy over this MDP can be learned from experience.

Source Task Creation for Curriculum Learning

This paper presents the more ambitious problem of curriculum learning in reinforcement learning, in which the goal is to design a sequence of source tasks for an agent to train on, such that final performance or learning speed is improved.

Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

This article presents a framework for curriculum learning (CL) in reinforcement learning, and uses it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals.

Curriculum learning

It is hypothesized that curriculum learning has both an effect on the speed of convergence of the training process to a minimum and on the quality of the local minima obtained: curriculum learning can be seen as a particular form of continuation method (a general strategy for global optimization of non-convex functions).

Curriculum-guided Hindsight Experience Replay

This paper proposes to adaptively select the failed experiences for replay according to the proximity to the true goals and the curiosity of exploration over diverse pseudo goals, and adopts a human-like learning strategy that enforces more curiosity in earlier stages and changes to larger goal-proximity later.

Learning Montezuma's Revenge from a Single Demonstration

A new method for learning from a single demonstration to solve hard exploration tasks like the Atari game Montezuma's Revenge, for which a trained agent achieving a high-score of 74,500 is presented, better than any previously published result.

Automatic Goal Generation for Reinforcement Learning Agents

This work uses a generator network to propose tasks for the agent to try to achieve, specified as goal states, and shows that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment.