# Adaptive Dynamic Programming: An Introduction

@article{Wang2009AdaptiveDP,
author={F. Wang and Huaguang Zhang and Derong Liu},
journal={IEEE Computational Intelligence Magazine},
year={2009},
volume={4}
}
• Published 2009
• Computer Science
• IEEE Computational Intelligence Magazine
In this article, we introduce some recent research trends within the field of adaptive/approximate dynamic programming (ADP), including the variations on the structure of ADP schemes, the development of ADP algorithms and applications of ADP schemes. For ADP algorithms, the point of focus is that iterative algorithms of ADP can be sorted into two classes: one class is the iterative algorithm with initial stable policy; the other is the one without the requirement of initial stable policy. It is… Expand
659 Citations

#### Figures and Topics from this paper

An Overview of Research on Adaptive Dynamic Programming
• Computer Science
• 2013
This paper gives a review of ADP in the order of the variation on the structure of ADp scheme, the development ofADP algorithms and applications, aiming to bring the reader into this novel field of optimization technology. Expand
• Computer Science
• IEEE Transactions on Systems, Man, and Cybernetics: Systems
• 2021
Through a comprehensive and complete investigation of its applications in many existing fields, this article fully demonstrates that the ADP intelligent control method is promising in today’s artificial intelligence era. Expand
• Computer Science
• 2017
This chapter reviews the development of adaptive dynamic programming (ADP). It starts with a background overview of reinforcement learning and dynamic programming. It then moves on to the basic formsExpand
A Summary on Some Typical Adaptive Dynamic Programming Schemes
• Computer Science
• 2018 37th Chinese Control Conference (CCC)
• 2018
This paper sums up four typical schemes of adaptive dynamic programming (ADP). The diagrams are provided and the algorithms of various schemes are described, which is convenient for comparison. SomeExpand
Robust adaptive dynamic programming: An overview of recent results
• Computer Science
• 2012
An application of robust-ADP to the decentralized optimal stabilization of large-scale systems is studied and an example of power systems is numerically simulated to validate the efficiency of the robust- ADP-based optimal control design. Expand
Adaptive Dynamic Programming - Discrete Version
• Computer Science
• 2018
This chapter presents the application of adaptive structures to the Bellman’s DP method to approximate the value function. Such action resulted in the creation of a family of neural dynamicExpand
A Supplementary Condition for the Convergence of the Control Policy during Adaptive Dynamic Programming
• Mathematics
• 2018
Reinforcement learning based adaptive/approximate dynamic programming (ADP) is a powerful technique to determine an approximate optimal controller for a dynamical system. These methods bypass theExpand
Dual iterative adaptive dynamic programming for a class of discrete-time nonlinear systems with time-delays
• Computer Science
• Neural Computing and Applications
• 2012
In this paper, a new dual iterative adaptive dynamic programming (ADP) algorithm is developed to solve optimal control problems for a class of nonlinear systems with time-delays in state and controlExpand
Adaptive dynamic programming with stable value iteration algorithm for discrete-time nonlinear systems
• Mathematics, Computer Science
• The 2012 International Joint Conference on Neural Networks (IJCNN)
• 2012
It is proved that any of iterative control obtained in the proposed algorithm can stabilize the nonlinear system which overcomes the disadvantage of traditional value iteration algorithms. Expand
A Novel Iterative $\theta$-Adaptive Dynamic Programming for Discrete-Time Nonlinear Systems
• Computer Science
• IEEE Transactions on Automation Science and Engineering
• 2014
It is proved that all the Iterative controls obtained in the iterative θ-ADP algorithm can stabilize the nonlinear system which means that the iteratives θ, which is feasible for implementations both online and offline, is feasible. Expand

#### References

SHOWING 1-10 OF 144 REFERENCES
A performance gradient perspective on approximate dynamic programming and its application to partially observable Markov decision processes
• Computer Science
• 2006 IEEE Conference on Computer Aided Control System Design, 2006 IEEE International Conference on Control Applications, 2006 IEEE International Symposium on Intelligent Control
• 2006
This paper shows an approach to integrating common approximate dynamic programming (ADP) algorithms into a theoretical framework to address both analytical characteristics and algorithmic features.Expand
Model-free Approximate Dynamic Programming Schemes for Linear Systems
• Computer Science, Mathematics
• 2007 International Joint Conference on Neural Networks
• 2007
Online model-free adaptive critic schemes based on approximate dynamic programming (ADP) to solve optimal control problems in both discrete-time and continuous-time domains for linear systems with unknown dynamics are presented. Expand
Direct Neural Dynamic Programming
• Computer Science
• 2004
This chapter discusses the relationships, results, and challenges of various approaches under the theme of ADP, and introduces the fundamental principles of the direct neural dynamic programming (NDP), which is demonstrated for a continuous state control problem using an industrial scale Apache helicopter model. Expand
• Mathematics, Computer Science
• IEEE Trans. Syst. Man Cybern. Part C
• 2002
An adaptive dynamic programming algorithm (ADPA) is described which fuses soft computing techniques to learn the optimal cost functional for a stabilizable nonlinear system with unknown dynamics and hard Computing techniques to verify the stability and convergence of the algorithm. Expand
A Novel Infinite-Time Optimal Tracking Control Scheme for a Class of Discrete-Time Nonlinear Systems via the Greedy HDP Iteration Algorithm
• Computer Science, Medicine
• IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)
• 2008
In this paper, we aim to solve the infinite-time optimal tracking control problem for a class of discrete-time nonlinear systems using the greedy heuristic dynamic programming (HDP) iterationExpand
Continuous-Time ADP for Linear Systems with Partially Unknown Dynamics
• Mathematics
• 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning
• 2007
Approximate dynamic programming has been formulated and applied mainly to discrete-time systems. Expressing the ADP concept for continuous-time systems raises difficult issues related to samplingExpand
Discrete-Time Nonlinear HJB Solution Using Approximate Dynamic Programming: Convergence Proof
• Mathematics, Computer Science
• IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)
• 2008
It is shown that HDP converges to the optimal control and the optimal value function that solves the Hamilton-Jacobi-Bellman equation appearing in infinite-horizon discrete-time (DT) nonlinear optimal control. Expand
Neural-Network-Based Near-Optimal Control for a Class of Discrete-Time Affine Nonlinear Systems With Control Constraints
• Mathematics, Computer Science
• IEEE Transactions on Neural Networks
• 2009
In this paper, the near-optimal control problem for a class of nonlinear discrete-time systems with control constraints is solved by iterative adaptive dynamic programming algorithm. First, a novelExpand
Handbook of Learning and Approximate Dynamic Programming
• Computer Science
• IEEE Transactions on Automatic Control
• 2006
This chapter discusses reinforcement learning in large, high-dimensional state spaces, model-based adaptive critic designs, and applications of approximate dynamic programming in power systems control. Expand
Model-free multiobjective approximate dynamic programming for discrete-time nonlinear systems with general performance index functions
• Mathematics, Computer Science
• Neurocomputing
• 2009
In this paper, a forward-in-time optimal control method for a class of discrete-time nonlinear systems with general multiobjective performance indices is proposed with unknown system dynamics. TheExpand