Performance Loss Bounds for Approximate Value Iteration with State Aggregation

@article{Roy2006PerformanceLB,
  title={Performance Loss Bounds for Approximate Value Iteration with State Aggregation},
  author={Benjamin Van Roy},
  journal={Math. Oper. Res.},
  year={2006},
  volume={31},
  pages={234-244}
}
We consider approximate value iteration with a parameterized approximator in which the state space is partitioned and the optimal cost-to-go function over each partition is approximated by a constant. We establish performance loss bounds for policies derived from approximations associated with fixed points. These bounds identify benefits to using invariant distributions of appropriate policies as projection weights. Such projection weighting relates to what is done by temporal-difference… Expand
92 Citations
TD(0) Leads to Better Policies than Approximate Value Iteration
  • 2
  • PDF
RAAM: The Benefits of Robustness in Approximating Aggregated MDPs in Reinforcement Learning
  • 18
  • Highly Influenced
  • PDF
Algorithms and Bounds for Sampling-based Approximate Policy Iteration
  • 6
Efficient Exploration and Value Function Generalization in Deterministic Systems
  • 42
  • PDF
State aggregation based linear programming approach to approximate dynamic programming
  • 25
  • PDF
Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Models
  • 16
  • Highly Influenced
  • PDF
Efficient Reinforcement Learning in Deterministic Systems with Value Function Generalization
  • 25
  • PDF
State partitioning based linear program for stochastic dynamic programs: An invariance property
  • 4
Feature-based aggregation and deep reinforcement learning: a survey and some new implementations
  • D. Bertsekas
  • Computer Science, Mathematics
  • IEEE/CAA Journal of Automatica Sinica
  • 2019
  • 70
  • Highly Influenced
  • PDF
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 60 REFERENCES
An upper bound on the loss from approximate optimal-value functions
  • 120
  • PDF
Stable Function Approximation in Dynamic Programming
  • 567
  • Highly Influential
  • PDF
Least Squares Policy Evaluation Algorithms with Linear Function Approximation
  • 202
  • Highly Influential
  • PDF
The Linear Programming Approach to Approximate Dynamic Programming
  • 681
  • PDF
An Iterative Aggregation Procedure for Markov Decision Processes
  • 60
Approximations of Dynamic Programs, I
  • W. Whitt
  • Mathematics, Computer Science
  • Math. Oper. Res.
  • 1978
  • 205
  • Highly Influential
Approximate Dynamic Programming via Linear Programming
  • 25
  • PDF
Analysis of Temporal-Diffference Learning with Function Approximation
  • 1,264
  • PDF
...
1
2
3
4
5
...