Online natural gradient as a Kalman filter

@article{Ollivier2017OnlineNG,
  title={Online natural gradient as a Kalman filter},
  author={Y. Ollivier},
  journal={Electronic Journal of Statistics},
  year={2017},
  volume={12},
  pages={2930-2961}
}
  • Y. Ollivier
  • Published 2017
  • Computer Science, Mathematics
  • Electronic Journal of Statistics
We establish a full relationship between Kalman filtering and Amari's natural gradient in statistical learning. Namely, using an online natural gradient descent on data log-likelihood to evaluate the parameter of a probabilistic model from a series of observations, is exactly equivalent to using an extended Kalman filter to estimate the parameter (assumed to have constant dynamics). In the i.i.d. case, this relation is a consequence of the "information filter" phrasing of the extended Kalman… Expand
View PDF on arXiv
34 Citations
Highly Influential Citations
1
Background Citations
10
Methods Citations
9
Results Citations
1

34 Citations

Citation Type

Citation Type

The Extended Kalman Filter is a Natural Gradient Descent in Trajectory Space
  • 7
  • PDF
The decoupled extended Kalman filter for dynamic exponential-family factorization models
  • 3
  • PDF
Gradient Structure Of The Ensemble Kalman Flow With Noise
  • 9
  • PDF
A unified theory of adaptive stochastic gradient descent as Bayesian filtering
  • 5
Iterative Nonlinear Kalman Filtering via Variational Evidence Lower Bound Maximization
Interacting Langevin Diffusions: Gradient Structure and Ensemble Kalman Sampler
  • 43
  • PDF
True Asymptotic Natural Gradient Optimization
  • 7
  • PDF
On Kalman-Bucy filters, linear quadratic control and active inference
  • 2
  • PDF
Learning-Algorithms from Bayesian Principles
  • 4
  • PDF
A Probabilistic Incremental Proximal Gradient Method
  • PDF
...
1
2
3
4
...

References

SHOWING 1-10 OF 37 REFERENCES
Hierarchical Bayesian Models for Regularization in Sequential Learning
  • 34
  • PDF
Training recurrent networks using the extended Kalman filter
  • R. J. Williams
  • Computer Science
  • [Proceedings 1992] IJCNN International Joint Conference on Neural Networks
  • 1992
  • 117
Training recurrent networks online without backtracking
  • 40
  • PDF
Information Geometric Approach to Recursive Update in Nonlinear Filtering
  • 8
  • PDF
Dual Kalman Filtering Methods for Nonlinear Prediction, Smoothing and Estimation
  • 85
  • PDF
Kalman-Based Stochastic Gradient Method with Stop Condition and Insensitivity to Conditioning
  • V. Patel
  • Mathematics, Computer Science
  • SIAM J. Optim.
  • 2016
  • 30
  • PDF
Bayesian Filtering and Smoothing
  • S. Särkkä
  • Computer Science
  • Institute of Mathematical Statistics textbooks
  • 2013
  • 917
  • PDF
Convergence analysis of the extended Kalman filter used as an observer for nonlinear deterministic discrete-time systems
  • 252
  • PDF
Natural Gradient Works Efficiently in Learning
  • S. Amari
  • Computer Science, Mathematics
  • Neural Computation
  • 1998
  • 2,486
Comparative Analysis of Backpropagation and the Extended Kalman Filter for Training Multilayer Perceptrons
  • 115
...
1
2
3
4
...