Deep Reinforcement Learning With Quantum-Inspired Experience Replay

@article{Wei2022DeepRL,
  title={Deep Reinforcement Learning With Quantum-Inspired Experience Replay},
  author={Qing Wei and Hailan Ma and Chunlin Chen and D. Dong},
  journal={IEEE Transactions on Cybernetics},
  year={2022},
  volume={52},
  pages={9326-9338}
}
In this article, a novel training paradigm inspired by quantum computation is proposed for deep reinforcement learning (DRL) with experience replay. In contrast to the traditional experience replay mechanism in DRL, the proposed DRL with quantum-inspired experience replay (DRL-QER) adaptively chooses experiences from the replay buffer according to the complexity and the replayed times of each experience (also called transition), to achieve a balance between exploration and exploitation. In DRL… 

Figures and Tables from this paper

Improving Experience Replay through Modeling of Similar Transitions' Sets

A new reinforcement learning method, COMPact Experience Replay (COMPER), which uses temporal difference learning with predicted target values based on recurrence over sets of similar transitions, and a new approach for experience replay based on two transitions memories is proposed and evaluated.

Asynchronous Curriculum Experience Replay: A Deep Reinforcement Learning Approach for UAV Autonomous Motion Control in Unknown Dynamic Environments

The autonomous motion control (AMC) problem is formulated as a Markov decision process (MDP) and an advanced deep reinforcement learning (DRL) method that allows UAVs to execute complex tasks in large-scale dynamic three-dimensional (3D) environments is proposed.

A Survey on Quantum Reinforcement Learning

A birds-eye-view of the literature on quantum reinforcement learning is provided, with a focus on already available noisy intermediate-scale quantum devices that include variational quantum circuits acting as function approximators in an otherwise classical reinforcement learning setting.

Variational Quantum Soft Actor-Critic

This work develops a quantum reinforcement learning algorithm based on soft actor-critic, a hybrid quantum-classical policy network consisting of a variational quantum circuit and a classical artificial neural network, and analyzes the effect of different hyper-parameters and policy network architectures.

A Sample Efficiency Improved Method via Hierarchical Reinforcement Learning Networks

This paper proposes a method to improve the sample efficiency of existing learning from demonstration approaches via data augmentation, dynamic experience replay sizes, and hierarchical Deep Q-Networks (DQN).

Path Planning for Cellular-Connected UAV: A DRL Solution with Quantum-Inspired Experience Replay

To help the DRL agent commit a better trade-off between sampling priority and diversity, a novel quantum-inspired experience replay (QiER) framework is proposed, via relating experienced transition’s importance to its associated quantum bit (qubit) and applying Grover iteration based amplitude amplification technique.

Quantum Language Model with Entanglement Embedding for Question Answering

This work proposes a NN model with a novel entanglement embedding (EE) module, whose function is to transform the word sequence into an entangled pure state representation, and shows that QLM-EE achieves superior performance compared with the classical deep NN models and other QLMs on question answering (QA) datasets.

Parallel Deep Reinforcement Learning Method for Gait Control of Biped Robot

A parallel Deep Deterministic Policy Gradient algorithm is presented for biped robot gait control and a Markov Decision Process (MDP) model based on knowledge and experience is designed to solve the problem of sparse rewards.

RIS-Assisted Multi-Antenna AmBC Signal Detection Using Deep Reinforcement Learning

An efficient multi-antenna AmBC system is developed based on RIS, which can achieve information transmission and energy collection simultaneously and a smart twin delayed deep deterministic (TD3) AmBC signal detection method is presented, based on deep reinforcement learning.

Crypto Makes AI Evolve

The path and stages of this evolution of AI, focusing on the role of quantum-inspired and bio-inspired AI, are studied, including Crypto-Sensitive AI, Crypto-Adapted AI, crypto-Friendly AI , Crypto-Enabled AI, and Crypto-Protected AI.

References

SHOWING 1-10 OF 56 REFERENCES

Quantum Reinforcement Learning

Inspired by the state superposition principle and quantum parallelism, a framework of value updating algorithm is introduced and the state (action) in traditional RL is identified as the eigen state (eigen action) in QRL.

Self-Paced Prioritized Curriculum Learning With Coverage Penalty in Deep Reinforcement Learning

The experimental results show that the proposed curriculum training paradigm of DCRL is also applicable and effective for other memory-based deep reinforcement learning approaches, such as double DQN and dueling network.

Competitive Experience Replay

This work proposes a novel method called competitive experience replay, which efficiently supplements a sparse reward by placing learning in the context of an exploration competition between a pair of agents, creating a competitive game designed to drive exploration.

Remember and Forget for Experience Replay

Remember and Forget Experience Replay (ReF-ER) is introduced, a novel method that can enhance RL algorithms with parameterized policies and consistently improves the performance of continuous-action, off-policy RL on fully observable benchmarks and partially observable flow control problems.

Attentive Experience Replay

Attentive Experience Replay is introduced, a novel experience replay algorithm that samples transitions according to the similarities between their states and the agent's state and it is demonstrated that AER makes consistent improvements on the suite of OpenAI gym tasks.

Deep Reinforcement Learning with Double Q-Learning

This paper proposes a specific adaptation to the DQN algorithm and shows that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.

Quantum-enhanced machine learning

This work proposes an approach for the systematic treatment of machine learning, from the perspective of quantum information, and shows that quadratic improvements in learning efficiency, and exponential improvements in performance over limited time periods, can be obtained for a broad class of learning problems.

Fidelity-Based Probabilistic Q-Learning for Control of Quantum Systems

A fidelity-based probabilistic Q-learning (FPQL) approach is presented to naturally solve this problem and applied for learning control of quantum systems and shows that FPQL algorithms attain a better balance between exploration and exploitation, and can also avoid local optimal policies and accelerate the learning process.

The importance of experience replay database composition in deep reinforcement learning

The potential of the Deep Deterministic Policy Gradient method for a robot control problem both in simulation and in a real setup is investigated and some requirements on the distribution over the state-action space of the experiences in the database are identified.

Human-level control through deep reinforcement learning

This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
...