Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

@article{Bhaya2004SteepestDW,
  title={Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method},
  author={Amit Bhaya and Eugenius Kaszkurewicz},
  journal={Neural networks : the official journal of the International Neural Network Society},
  year={2004},
  volume={17 1},
  pages={65-71}
}
It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system. 
Highly Cited
This paper has 99 citations. REVIEW CITATIONS

Citations

Publications citing this paper.
Showing 1-10 of 28 extracted citations

Convergence of gradient method with momentum for two-Layer feedforward neural networks

IEEE Transactions on Neural Networks • 2006
View 3 Excerpts
Highly Influenced

A Simple Yet Efficient Evolution Strategy for Large-Scale Black-Box Optimization

IEEE Transactions on Evolutionary Computation • 2018
View 1 Excerpt

99 Citations

01020'06'09'12'15'18
Citations per Year
Semantic Scholar estimates that this publication has 99 citations based on the available data.

See our FAQ for additional information.

References

Publications referenced by this paper.
Showing 1-10 of 22 references

Iterative Methods for Solving Linear Systems of Equations on FPGA-Based Machines

Computers and Their Applications • 2003
View 4 Excerpts
Highly Influenced

Stability of steepest descent with momentum for quadratic functions

IEEE Trans. Neural Networks • 2002
View 8 Excerpts
Highly Influenced

The heavy ball with friction method, I. The continuous dynamical system: global exploration of the global minima of a real-valued function by asymptotic analysis of a dissipative dynamical system

H. Attouch, X. Goudou, P. Redont
Communications in Contemporary Mathematics, • 2000
View 7 Excerpts
Highly Influenced

On the momentum term in gradient descent learning algorithms

Neural Networks • 1999
View 13 Excerpts
Highly Influenced

Analysis of Momentum Term in Back-Propagation

IEICE Transactions • 1995
View 5 Excerpts
Highly Influenced

Analysis of the back-propagation algorithm with momentum

IEEE Trans. Neural Networks • 1994
View 4 Excerpts
Highly Influenced

Continuous processes of the Newton type

Y. I. Alber
Differential Equations • 1971
View 3 Excerpts
Highly Influenced

Control system design

G. C. Goodwin, S. F. Graebe, M. E. Salgado
Upper Saddle River, NJ: Prentice-Hall • 2001

Neural networks (2nd ed.)

S. Haykin
Upper Saddle River, NJ: Prentice-Hall • 1999
View 2 Excerpts

Similar Papers

Loading similar papers…