# Acceleration of stochastic approximation by averaging

@article{Polyak1992AccelerationOS, title={Acceleration of stochastic approximation by averaging}, author={Boris Polyak and Anatoli B. Juditsky}, journal={Siam Journal on Control and Optimization}, year={1992}, volume={30}, pages={838-855} }

A new recursive algorithm of stochastic approximation type with the averaging of trajectories is investigated. Convergence with probability one is proved for a variety of classical optimization and identification problems. It is also demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence.

## 1,716 Citations

### Asymptotically efficient stochastic approximation

- Mathematics, Computer Science
- 1993

Asymptotic efficiency of a stochastic approximation algorithm is proved under weak conditions: no growth rate restriction is imposed on the regression function; wide class of observation noises is…

### On Stochastic Approximation Procedures with Averaging

- Mathematics
- 2000

Linear multidimensional stochastic approximation procedures in continuous time with martingale errors are considered. An asymptotic behavior of the estimator obtained by trajectory averaging is…

### Asymptotic Expansions for the Stochastic Approximation Averaging Procedure in Continuous Time

- Mathematics
- 1998

This paper considers the statistical estimation problem of the root of a nonlinear function under observations of the nonlinear regression model in continuous time. Asymptotic expansions for…

### Quasi stochastic approximation

- Computer ScienceProceedings of the 2011 American Control Conference
- 2011

A general foundation for "quasi-stochastic approximation" in which all of the processes under consideration are deterministic, much like quasi-Monte-Carlo for variance reduction in simulation is provided.

### Asymptotic behavior of asynchronous stochastic approximation

- Computer ScienceScience in China Series : Information Sciences
- 2007

The pathwise convergence of a distributed, asynchronous stochastic approximation (SA) scheme is analyzed. The conditions imposed on the step size and noise are the weakest in comparison with the…

### Deviation inequalities for stochastic approximation by averaging

- MathematicsStochastic Processes and their Applications
- 2022

### Rates of Convergence of Adaptive Step-Size of Stochastic Approximation Algorithms

- Computer Science, Mathematics
- 2000

A new adaptive algorithm with decreasing step-size for stochastic approximations with optimality of the rate of convergence is proposed and an averaging algorithm is used to control the step sizes.

### Globally convergent stochastic optimization with optimal asymptotic distribution

- Mathematics, Computer ScienceJournal of Applied Probability
- 1998

A stochastic gradient descent method is combined with a consistent auxiliary estimate to achieve global convergence of the recursion and the optimal convergence rate and variance of the asymptotic distribution.

### Martingale Models of Stochastic Approximation and Their Convergence

- Mathematics
- 2000

Procedures of stochastic approximation are studied from a general theory of stochastic processes point of view. The results on convergence are obtained by the uniform methods both in the case of…

### Some Limit Properties of Markov Chains Induced by Recursive Stochastic Algorithms

- Computer ScienceSIAM J. Math. Data Sci.
- 2020

Recursive stochastic algorithms have gained significant attention in the recent past due to data-driven applications and can be used for solving large-scale optimizati...

## References

SHOWING 1-10 OF 20 REFERENCES

### Stochastic approximation method with gradient averaging for unconstrained problems

- Mathematics, Computer Science
- 1983

A stochastic approximation algorithm is studied, in which random gradient estimates are averaged by an auxiliary, filter. Convergence of the algorithm is proved under a rather general noise…

### Asymptotically efficient stochastic approximation

- Mathematics, Computer Science
- 1993

Asymptotic efficiency of a stochastic approximation algorithm is proved under weak conditions: no growth rate restriction is imposed on the regression function; wide class of observation noises is…

### Convergence analysis of smoothed stochastic gradient-type algorithm

- Computer Science
- 1987

A rigorous analysis is presented which concludes, under very mild assumptions on the data, that the smoothed stochastic gradient (SSG) algorithm converges almost everywhere.

### Theory and Practice of Recursive Identification

- Computer Science
- 1983

Methods of recursive identification deal with the problem of building mathematical models of signals and systems on-line, at the same time as data is being collected. Such methods, which are also k…

### Adaptive filtering prediction and control

- Business
- 1984

This unified survey focuses on linear discrete-time systems and explores the natural extensions to nonlinear systems and summarizes the theoretical and practical aspects of a large class of adaptive algorithms.

### Stochastic Approximation and Recursive Estimation

- Art
- 1976

An extrusion head for an extrudable material, such as a food product dough, which is to be formed into a special shape, such as a pretzel configuration. The head includes a die and associated parts…