Corpus ID: 85459437

Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder

  title={Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder},
  author={Kim Heecheol and Masanori Yamada and Kosuke Miyoshi and Hiroshi Yamakawa},
One problem in the application of reinforcement learning to real-world problems is the curse of dimensionality on the action space. Macro actions, a sequence of primitive actions, have been studied to diminish the dimensionality of the action space with regard to the time axis. However, previous studies relied on humans defining macro actions or assumed macro actions as repetitions of the same primitive actions. We present Factorized Macro Action Reinforcement Learning (FaMARL) which… Expand
Construction of Macro Actions for Deep Reinforcement Learning
This work proposes to construct macro actions by a genetic algorithm, which eliminates the dependency of the macro action derivation procedure from the past policies of the agent, and performs extensive experiments to show that the constructed macro actions are able to speed up the learning process for a variety of deep reinforcement learning methods. Expand
  • 2019
In this paper, we propose to improve the performance of deep reinforcement learning (DRL) methods by searching for a feasible macro action ensemble to augment the action space of an agent. A macroExpand
Toward Synergism in Macro Action Ensembles
Macro actions have been demonstrated to be beneficial for the learning processes of an agent. A variety of techniques have been developed to construct more effective macro actions. However, theyExpand
Reusability and Transferability of Macro Actions for Reinforcement Learning.
This paper uncovers the properties of reusability and transferability of macro actions, which means that a macro action generated along with one RL method can be reused by another RL method for training, while the second one, transferability, means thatA macro action can be utilized for training agents in similar environments with different reward settings. Expand


Deep Reinforcement Learning With Macro-Actions
This paper focuses on macro-actions, and evaluates these on different Atari 2600 games, where they yield significant improvements in learning speed and can even achieve better scores than DQN. Expand
Deep Reinforcement Learning in Parameterized Action Space
This paper represents a successful extension of deep reinforcement learning to the class of parameterized action space MDPs within the domain of simulated RoboCup soccer, which features a small set of discrete action types each of which is parameterized with continuous variables. Expand
Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning
A novel framework, Fine Grained Action Repetition (FiGAR), which enables the agent to decide the action as well as the time scale of repeating it and can be used for improving any Deep Reinforcement Learning algorithm which maintains an explicit policy estimate by enabling temporal abstractions in the action space. Expand
Dynamic Action Repetition for Deep Reinforcement Learning
A new framework Dynamic Action Rep repetition is proposed which changes Action Repetition Rate (the time scale of repeating an action) from a hyper-parameter of an algorithm to a dynamically learnable quantity. Expand
Human-level control through deep reinforcement learning
This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks. Expand
Strategic Attentive Writer for Learning Macro-Actions
A novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting, which is at the same time a general algorithm that can be applied on any sequence data. Expand
FAVAE: Sequence Disentanglement using Information Bottleneck Principle
The factorized action variational autoencoder (FAVAE), a state-of-the-art generative model for learning disentangled and interpretable representations from sequential data via the information bottleneck without supervision, can disentangle multiple dynamic factors. Expand
Asynchronous Methods for Deep Reinforcement Learning
A conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers and shows that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input. Expand
Learning to Walk via Deep Reinforcement Learning
A sample-efficient deep RL algorithm based on maximum entropy RL that requires minimal per-task tuning and only a modest number of trials to learn neural network policies is proposed and achieves state-of-the-art performance on simulated benchmarks with a single set of hyperparameters. Expand
End-to-End Training of Deep Visuomotor Policies
This paper develops a method that can be used to learn policies that map raw image observations directly to torques at the robot's motors, trained using a partially observed guided policy search method, with supervision provided by a simple trajectory-centric reinforcement learning method. Expand