Corpus ID: 237605424

Generalisations and improvements of New Q-Newton's method Backtracking

@article{Truong2021GeneralisationsAI,
  title={Generalisations and improvements of New Q-Newton's method Backtracking},
  author={Tuyen Trung Truong},
  journal={ArXiv},
  year={2021},
  volume={abs/2109.11395}
}
  • T. Truong
  • Published 23 September 2021
  • Computer Science, Mathematics
  • ArXiv
In this paper, we propose a general framework for the algorithm New Q-Newton’s method Backtracking, developed in the author’s previous work. For a symmetric, square real matrix A, we define minsp(A) := min||e||=1 ||Ae||. Given a C 2 cost function f : R → R and a real number 0 < τ , as well as m+ 1 fixed real numbers δ0, . . . , δm, we define for each x ∈ R m with ∇f(x) 6= 0 the following quantities: κ := mini6=j |δi − δj |; A(x) := ∇f(x) + δ||∇f(x)|| Id, where δ is the first element in the… Expand

References

SHOWING 1-10 OF 15 REFERENCES
New Q-Newton's method meets Backtracking line search: good convergence guarantee, saddle points avoidance, quadratic rate of convergence, and easy implementation
  • T. Truong
  • Mathematics, Computer Science
  • ArXiv
  • 2021
TLDR
The convergence guarantee issue is resolved by proposing a modification of New Q-Newton’s method, named NewQ-Newtons method Backtracking, which incorporates a more sophisticated use of hyperparameters and a Backtracking line search, which has very good theoretical guarantees, which for a Morse function yields the following. Expand
Backtracking Gradient Descent Method and Some Applications in Large Scale Optimisation. Part 2: Algorithms and Experiments
In this paper, we provide new results and algorithms (including backtracking versions of Nesterov accelerated gradient and Momentum) which are more applicable to large scale optimisation as in DeepExpand
Convergence of the Iterates of Descent Methods for Analytic Cost Functions
TLDR
It is shown that the iterates of numerical descent algorithms, for an analytic cost function, share this convergence property if they satisfy certain natural descent conditions and strengthen classical "weak convergence" results for descent methods to "strong limit-point convergence" for a large class of cost functions of practical interest. Expand
Convergence to minima for the continuous version of Backtracking Gradient Descent
  • T. Truong
  • Mathematics, Computer Science
  • ArXiv
  • 2019
TLDR
The main result is that any cluster point of the sequence H(x)=H(x_n) is not a saddle point, and more generally cannot be an isolated generalised saddle point. Expand
Some convergent results for Backtracking Gradient Descent method on Banach spaces
TLDR
The main result concerns the following condition, which is that whenever whenever $\{x_n\}$ weakly converges to $x$ and $\lim _{n\rightarrow\infty}||\nabla f(x)|=0', then $f(x)=0$. Expand
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
TLDR
This paper proposes a new approach to second-order optimization, the saddle-free Newton method, that can rapidly escape high dimensional saddle points, unlike gradient descent and quasi-Newton methods, and applies this algorithm to deep or recurrent neural network training, and provides numerical evidence for its superior optimization performance. Expand
Statistics of critical points of Gaussian fields on large-dimensional spaces.
TLDR
The results give a complete picture of the organization of critical points and are of relevance to glassy and disordered systems and landscape scenarios coming from the anthropic approach to string theory. Expand
Unconstrained optimisation on Riemannian manifolds, arXiv:2008.11091
  • 2008
Explicit bounds for the Lojasiewicz exponent in the gradient inequality for polynomials, Ann
  • Pol. Math
  • 2005
...
1
2
...