• Corpus ID: 49211736

Automatic formation of the structure of abstract machines in hierarchical reinforcement learning with state clustering

  title={Automatic formation of the structure of abstract machines in hierarchical reinforcement learning with state clustering},
  author={Aleksandr I. Panov and Aleksey Skrynnik},
We introduce a new approach to hierarchy formation and task decomposition in hierarchical reinforcement learning. Our method is based on the Hierarchy Of Abstract Machines (HAM) framework because HAM approach is able to design efficient controllers that will realize specific behaviors in real robots. The key to our algorithm is the introduction of the internal or "mental" environment in which the state represents the structure of the HAM hierarchy. The internal action in this environment leads… 
1 Citations

Figures from this paper

Hierarchical Reinforcement Learning with Clustering Abstract Machines

This paper presents a new approach for simultaneous constructing and applying the hierarchy of actions and sub-goals based on a united loop of clustering of the environment’s states observed by the agent and allocation of sub-targets by the modified bottleneck method for constructing of abstract machines hierarchy.



Dynamic abstraction in reinforcement learning via clustering

This work considers a graph theoretic approach for automatic construction of options in a dynamic environment and considers building a map that includes preliminary indication of the location of "interesting" regions of the state space, where the value gradient is significant and additional exploration might be beneficial.

Reinforcement Learning with Hierarchies of Machines

This work presents provably convergent algorithms for problem-solving and learning with hierarchical machines and demonstrates their effectiveness on a problem with several thousand states.

Hierarchical Approaches

  • B. Hengst
  • Computer Science, Psychology
    Reinforcement Learning
  • 2012
This Chapters introduces hierarchical approaches to reinforcement learning that hold out the promise of reducing a reinforcement learning problems to a manageable size and algorithms that can automatically learn the hierarchical structure from interactions with the domain.

A neural model of hierarchical reinforcement learning

A novel, biologically detailed neural model of reinforcement learning (RL) processes in the brain that divides the RL process into a hierarchy of actions at different levels of abstraction is developed and demonstrates improved performance as a result of its hierarchical ability.

Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition

The paper presents an online model-free learning algorithm, MAXQ-Q, and proves that it converges with probability 1 to a kind of locally-optimal policy known as a recursively optimal policy, even in the presence of the five kinds of state abstraction.

Q-Cut - Dynamic Discovery of Sub-goals in Reinforcement Learning

The Q-Cut algorithm is presented, a graph theoretic approach for automatic detection of sub-goals in a dynamic environment, which is used for acceleration of the Q-Learning algorithm, and extended to the Segmented Q- cut algorithm, which uses previously identified bottlenecks for state space partitioning, necessary for finding additional bottlenECks in complex environments.

Probabilistic inference for determining options in reinforcement learning

The proposed approach is based on parametric option representations and works well in combination with current policy search methods, which are particularly well suited for continuous real-world tasks.

Cognitive Mapping and Planning for Visual Navigation

The Cognitive Mapper and Planner is based on a unified joint architecture for mapping and planning, such that the mapping is driven by the needs of the task, and a spatial memory with the ability to plan given an incomplete set of observations about the world.

Automatic discovery and transfer of MAXQ hierarchies

It is demonstrated empirically that HI-MAT constructs compact hierarchies that are comparable to manually-engineered hierarchies and facilitate significant speedup in learning when transferred to a target task.

Feudal Reinforcement Learning

This paper shows how to create a Q-learning managerial hierarchy in which high level managers learning how to set tasks to their submanagers who, in turn, learn how to satisfy them.