• DOI:10.48550/arXiv.2205.13044
• Corpus ID: 249097779

Near-Optimal Goal-Oriented Reinforcement Learning in Non-Stationary Environments

@article{Chen2022NearOptimalGR,
  title={Near-Optimal Goal-Oriented Reinforcement Learning in Non-Stationary Environments},
  author={Liyu Chen and Haipeng Luo},
  journal={ArXiv},
  year={2022},
  volume={abs/2205.13044}
}

• Liyu Chen, Haipeng Luo
• Published 25 May 2022
• Computer Science
• ArXiv

We initiate the study of dynamic regret minimization for goal-oriented reinforcement learning modeled by a non-stationary stochastic shortest path problem with changing cost and transition functions. We start by establishing a lower bound Ω(( B ⋆ SAT ⋆ (∆ c + B 2 ⋆ ∆ P )) 1 / 3 K 2 / 3 ) , where B ⋆ is the maximum expected cost of the optimal policy of any episode starting from any state, T ⋆ is the maximum hitting time of the optimal policy of any episode starting from the initial state, SA is the number of state-action pairs, ∆ c and ∆ P are the amount of changes of the cost and transition functions respectively, and K is the number of episodes. The different roles of ∆ c and ∆ P in this lower bound inspire us to design algorithms that estimate costs and transitions separately. Specifically, assuming the knowledge of ∆ c and ∆ P , we develop a simple but sub-optimal algorithm and another more involved minimax optimal algorithm (up to logarithmic terms). These algorithms combine the ideas of finite-horizon approximation [Chen et al., 2022a], special Bernstein-style bonuses of the MVP algorithm [Zhang et al., 2020], adaptive confidence widening [Wei and Luo, 2021], as well as some new techniques such as properly penalizing long-horizon policies. Finally, when ∆ c and ∆ P are unknown, we develop a variant of the MASTER algorithm [Wei and Luo, 2021] and integrate the aforementioned ideas into it to achieve ˜ O (min { B ⋆ S √ ALK, ( B 2 ⋆ S 2 AT ⋆ (∆ c + B ⋆ ∆ P )) 1 / 3 K 2 / 3 } ) regret, where L is the unknown number of changes of the environment.