# Equations of States in Singular Statistical Estimation

@article{Watanabe2010EquationsOS, title={Equations of States in Singular Statistical Estimation}, author={Sumio Watanabe}, journal={Neural networks : the official journal of the International Neural Network Society}, year={2010}, volume={23 1}, pages={ 20-34 } }

## Tables from this paper

## 62 Citations

A formula of equations of states in singular learning machines

- Mathematics2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence)
- 2008

A formula of equations of states is established which holds among Bayes and Gibbs generalization and training errors, and it is shown that two generalization errors can be estimated from two training errors.

Equations of States in Statistical Learning for a Nonparametrizable and Regular Case

- MathematicsArXiv
- 2009

It is proved that the same equations hold even if a true distribution is not contained in a parametric model, and the proposed equations in a regular case are asymptotically equivalent to the Takeuchi information criterion.

Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory

- Computer Science, MathematicsJ. Mach. Learn. Res.
- 2010

The Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable and model selection and hyperparameter optimization using these two values are asymPTOTically equivalent.

Statistical Learning Theory of Quasi-Regular Cases

- MathematicsIEICE Trans. Fundam. Electron. Commun. Comput. Sci.
- 2012

It is proved that, in a quasi-regular case, two birational invariants are equal to each other, resulting that the symmetry of the generalization and training errors holds and the quasi- regular case is useful to study statistical learning theory.

An Introduction to Algebraic Geometry and Statistical Learning Theory

- Mathematics, Computer Science
- 2012

In this book, an algebraic geometrical method is established on which the conventional statistical theory of regular statistical models does not hold, and it is theoretically shown that, in singular models, Bayes estimation is more appropriate than one point estimation, even asymptotically.

Accuracy of latent-variable estimation in Bayesian semi-supervised learning

- Computer ScienceNeural Networks
- 2015

Learning Coefficient of Generalization Error in Bayesian Estimation and Vandermonde Matrix-Type Singularity

- Computer ScienceNeural Computation
- 2012

This letter gives tight new bound values of learning coefficients for Vandermonde matrix-type singularities and the explicit values with certain conditions, which can show the learning coefficients of three-layered neural networks and normal mixture models.

Asymptotic accuracy of distribution-based estimation of latent variables

- Computer ScienceJ. Mach. Learn. Res.
- 2014

The present paper formulates distribution-based functions for the errors in the estimation of the latent variables of hierarchical statistical models and analyzes the asymptotic behavior for both the maximum likelihood and the Bayes methods.

Asymptotic Learning Curve and Renormalizable Condition in Statistical Learning Theory

- MathematicsArXiv
- 2010

This paper defines a renormalizable condition of the statistical estimation problem, and shows that, under such a condition, the asymptotic learning curves are ensured to be subject to the universal law, even if the true distribution is unrealizable and singular for a statistical model.

A Limit Theorem in Singular Regression Problem

- MathematicsArXiv
- 2009

A limit theorem is proved which shows the relation between the singular regression problem and two birational invariants, a real log canonical threshold and a singular fluctuation and enables us to estimate the generalization error from the training error without any knowledge of the true probability distribution.

## References

SHOWING 1-10 OF 34 REFERENCES

Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities

- MathematicsNeural Computation
- 2003

The Bayes generalization error is studied under the condition that the Kullback distance of the true distribution from the distribution represented by singularities is in proportion to 1/n and two results are shown.

Singularities in mixture models and upper bounds of stochastic complexity

- Computer Science, MathematicsNeural Networks
- 2003

Learning efficiency of redundant neural networks in Bayesian estimation

- Computer Science, MathematicsIEEE Trans. Neural Networks
- 2001

This paper proves that the Bayesian stochastic complexity of a layered neural network is asymptotically smaller than that of a regular statistical model if it contains the true distribution. Weâ€¦

Algebraic geometrical methods for hierarchical learning machines

- Computer ScienceNeural Networks
- 2001

Exchange Monte Carlo Sampling From Bayesian Posterior for Singular Learning Machines

- Computer ScienceIEEE Transactions on Neural Networks
- 2008

The idea that the exchange MC method has a better effect on Bayesian learning in singular learning machines than that in regular learning machines is proposed, and its effectiveness is shown by comparing the numerical stochastic complexity with the theoretical one.

Algebraic Analysis for Singular Statistical Estimation

- Mathematics, Computer ScienceALT
- 1999

This paper clarifies learning efficiency of a non-regular parametric model such as a neural network whose true parameter set is an analytic variety with singular points. By using Sato's b-function weâ€¦

Algebraic geometry of singular learning machines and symmetry of generalization and training errors

- Computer ScienceNeurocomputing
- 2005

Algebraic Analysis for Nonidentifiable Learning Machines

- Computer Science, MathematicsNeural Computation
- 2001

It is rigorously proved that the Bayesian stochastic complexity or the free energy is asymptotically equal to 1 logn (m1 1) loglogn + constant, where n is the number of training samples and 1 and m1 are the rational number and the natural number, which are determined as the birational invariant values of the singularities in the parameter space.

Algebraic Information Geometry for Learning Machines with Singularities

- Computer ScienceNIPS
- 2000

The rigorous asymptotic form of the stochastic complexity is clarified based on resolution of singularities and two different problems are studied.

On the Problem in Model Selection of Neural Network Regression in Overrealizable Scenario

- Computer ScienceNeural Computation
- 2002

The article analyzes the expected training error and the expected generalization error of neural networks and radial basis functions in overrealizable cases and clarifies the difference from regular models, for which identifiability holds.