Stochastic Approximation and Recursive Algorithms and Applications

@inproceedings{Kushner2003StochasticAA,
  title={Stochastic Approximation and Recursive Algorithms and Applications},
  author={Harold J. Kushner and George Yin},
  year={2003}
}
Introduction 1 Review of Continuous Time Models 1.1 Martingales and Martingale Inequalities 1.2 Stochastic Integration 1.3 Stochastic Differential Equations: Diffusions 1.4 Reflected Diffusions 1.5 Processes with Jumps 2 Controlled Markov Chains 2.1 Recursive Equations for the Cost 2.2 Optimal Stopping Problems 2.3 Discounted Cost 2.4 Control to a Target Set and Contraction Mappings 2.5 Finite Time Control Problems 3 Dynamic Programming Equations 3.1 Functionals of Uncontrolled Processes 3.2… 
Regime Switching Stochastic Approximation Algorithms with Application to Adaptive Discrete Stochastic Optimization
TLDR
By a combined use of the SA method and two-time-scale Markov chains, asymptotic properties of the algorithm are obtained, which are distinct from the usual SA techniques.
Stochastic Approximation with Markov Noise: Analysis and applications in reinforcement learning
TLDR
An asymptotic convergence analysis of two time-scale stochastic approximation driven by "controlled" Markov noise is presented for the first time and is shown to be useful in analyzing the tracking ability of general "adaptive" algorithms.
Stochastic Approximation Algorithms in the Estimation of Quasi-Stationary Distribution of Finite and General State Space Markov Chains
  • S. Zheng
  • Computer Science, Mathematics
  • 2014
TLDR
This thesis studies stochastic approximation algorithms for estimating the quasi-stationary distribution of Markov chains and extends the ODE method of [21] and proves the consistency of algorithm.
Least mean square algorithms with Markov regime-switching limit
TLDR
This work analyzes the tracking performance of the least mean square (LMS) algorithm for adaptively estimating a time varying parameter that evolves according to a finite state Markov chain and derives the limit dynamics satisfied by continuous-time interpolation of the estimates.
Stochastic approximation: a survey
TLDR
Extensions of stochastic recursive algorithms include multiple time scales, tracking of time changing systems, state‐dependent noise, rate of convergence, and random direction methods for high‐dimensional problems.
Discretization of processes at stopping times and Uncertainty quantification of stochastic approximation limits
TLDR
This work provides a model-adaptive asymptotically optimal discretization strategy that does not require any prior knowledge about the model, and introduces the so-called Uncertainty for SA (USA) algorithm, an SA algorithm in increasing dimension for computing the basis coefficients of a chaos expansion of this function on an orthogonal basis of a suitable Hilbert space.
Some Limit Properties of Markov Chains Induced by Stochastic Recursive Algorithms
TLDR
This work considers iterated random operators over a Polish space that simulate iterated contraction operator over that Polish space, and shows that under certain conditions, the time average of the random sequence converges to the spatial mean of the invariant distribution.
Trajectory averaging for stochastic approximation MCMC algorithms
  • F. Liang
  • Computer Science, Mathematics
  • 2010
TLDR
It is shown that the trajectory averaging estimator is asymptotically efficient for the stochastic approximation MCMC (SAMCMC) algorithm under mild conditions, and then this result is applied to the stoChastic approximation Monte Carlo algorithm.
Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noise
TLDR
The bounds show that there is no discrepancy in the convergence rate between Markovian and martingale noise, only the constants are affected by the mixing time of the Markov chain.
...
...