Corpus ID: 155091694

Information criteria for non-normalized models

  title={Information criteria for non-normalized models},
  author={Takeru Matsuda and Masatoshi Uehara and Aapo Hyv{\"a}rinen},
Many statistical models are given in the form of non-normalized densities with an intractable normalization constant. Since maximum likelihood estimation is computationally intensive for these models, several estimation methods have been developed which do not require explicit computation of the normalization constant, such as noise contrastive estimation (NCE) and score matching. However, model selection methods for general non-normalized models have not been proposed so far. In this study, we… Expand
On Selection Criteria for the Tuning Parameter in Robust Divergence
A selection criterion based on an asymptotic approximation of the Hyvarinen score applied to an unnormalized model defined by robust divergence is proposed, which demonstrates the usefulness of the proposed method via numerical studies using normal distributions and regularized linear regression. Expand
General Bayesian Loss Function Selection and the use of Improper Models
Statisticians often face the choice between using probability models or a paradigm defined by minimising a loss function. Both approaches are useful and, if the loss can be re-cast into a properExpand


Estimation of Non-Normalized Statistical Models by Score Matching
  • A. Hyvärinen
  • Computer Science, Mathematics
  • J. Mach. Learn. Res.
  • 2005
While the estimation of the gradient of log-density function is, in principle, a very difficult non-parametric problem, it is proved a surprising result that gives a simple formula that simplifies to a sample average of a sum of some derivatives of the log- density given by the model. Expand
A Unified Statistically Efficient Estimation Framework for Unnormalized Models
A unified, statistically efficient estimation framework for unnormalized models and several efficient estimators, whose asymptotic variance is the same as the MLE, is proposed. Expand
Estimation of Non-Normalized Mixture Models
The proposed method provides a probabilistically principled clustering method that is able to utilize a deep representation and applications to clustering of natural images and neuroimag- ing data give promising results. Expand
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
A new estimation principle is presented to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise, using the model log-density function in the regression nonlinearity, which leads to a consistent (convergent) estimator of the parameters. Expand
Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
The basic idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise and it is shown that the new method strikes a competitive trade-off in comparison to other estimation methods for unnormalized models. Expand
Some extensions of score matching
  • A. Hyvärinen
  • Mathematics, Computer Science
  • Comput. Stat. Data Anal.
  • 2007
It is shown how to estimate non-normalized models defined in the non-negative real domain, i.e. R"+^n", and it is shown that the score matching estimator can be obtained in closed form for some exponential families. Expand
Analysis of Noise Contrastive Estimation from the Perspective of Asymptotic Variance
This paper proposes a method for reducing asymptotic variance by estimating the parameters of the auxiliary distribution of NCE, and determines the form of the objective functions, where the asymPTotic variance takes the smallest values in the original estimator class and the proposed estimator classes. Expand
Generalized Score Matching for Non-Negative Data
This paper gives a generalized form of score matching for non-negative data that improves estimation efficiency and addresses an overlooked inexistence problem by generalizing the regularized score matching method of Lin et al. (2016) and improving its theoretical guarantees fornon-negative Gaussian graphical models. Expand
Bayesian Model Comparison with the Hyvärinen Score: Computation and Consistency
ABSTRACT The Bayes factor is a widely used criterion in model comparison and its logarithm is a difference of out-of-sample predictive scores under the logarithmic scoring rule. However, when some ofExpand
Imputation estimators for unnormalized models with missing data
Simulation results with truncated Gaussian graphical models and the application to real data of wind direction reveal that the proposed methods effectively enable statistical inference with unnormalized models from missing data. Expand