Tensor decompositions for learning latent variable models

@article{Anandkumar2014TensorDF,
  title={Tensor decompositions for learning latent variable models},
  author={Anima Anandkumar and Rong Ge and Daniel J. Hsu and Sham M. Kakade and Matus Telgarsky},
  journal={ArXiv},
  year={2014},
  volume={abs/1210.7559}
}
This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models--including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation--which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the… 
Tensor Decompositions for Learning Latent Variable Models (A Survey for ALT)
TLDR
This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models--including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation--which exploits a certain tensor structure in their low-order observable moments typically, of second- and third-order.
The Application of Symmetric Tensor Decomposition to Parameter Estimation of Latent Variable Models
TLDR
This paper analyzes the application of two symmetric tensor decomposition methods to parameter estimation of certain latent variable models and examines the method of orthogonalization proposed by Anandkumar et al. and implemented through the tensor power method.
Nonparametric Estimation of Multi-View Latent Variable Models
TLDR
A kernel method for learning multi-view latent variable models, allowing each mixture component to be nonparametric, and then the latent parameters are recovered using a robust tensor power method.
Smoothed analysis of tensor decompositions
TLDR
This work introduces a smoothed analysis model for studying generative models and develops an efficient algorithm for tensor decomposition in the highly overcomplete case (rank polynomial in the dimension) and shows that tensor products of perturbed vectors are linearly independent in a robust sense.
Learning Negative Mixture Models by Tensor Decompositions
TLDR
This work proposes a method to estimate the parameters of negative mixture models having a specific tensor structure in their low order observable moments, and introduces a generalization of the tensor power method for complex valued tensors, and establishes theoretical convergence guarantees.
CP Factor Model for Dynamic Tensors
TLDR
A new high order projection estimator is proposed for such a factor model, utilizing the special structure and the idea of the higher order orthogonal iteration procedures commonly used in Tucker-type tensor factor model and general tensor CP decomposition procedures.
Discovery of Latent Factors in High-dimensional Data Using Tensor Methods
TLDR
This is the first work that gives global convergence guarantees for the stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points and large-scale deployment of spectral methods (matrix and tensor decomposition) carried out on CPU, GPU and Spark platforms.
Latent Dirichlet Allocation on Spark via Tensor Decomposition
TLDR
This work proposes a framework to overcome the problem of unscalable and non-convex likelihood by taking the power of inverse method of moments and incorporates dimensionality reduction and thus is scalable to high-dimensional data.
Uniqueness of Tensor Decompositions with Applications to Polynomial Identifiability
TLDR
It is proved that given a tensor whose decomposition satisfies a robust form of Kruskal's rank condition, it is possible to approximately recover the decomposition if the tensor is known up to a sufficiently small (inverse polynomial) error.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 103 REFERENCES
Smoothed analysis of tensor decompositions
TLDR
This work introduces a smoothed analysis model for studying generative models and develops an efficient algorithm for tensor decomposition in the highly overcomplete case (rank polynomial in the dimension) and shows that tensor products of perturbed vectors are linearly independent in a robust sense.
A Spectral Algorithm for Latent Dirichlet Allocation
TLDR
This work provides a simple and efficient learning procedure that is guaranteed to recover the parameters for a wide class of multi-view models and topic models, including latent Dirichlet allocation (LDA).
Identifiability of parameters in latent structure models with many observed variables
TLDR
A general approach for establishing identifiability utilizing algebraic arguments is demonstrated, which sheds light on the properties of finite mixtures of Bernoulli products, which have been used for decades despite being known to have nonidentifiable parameters.
A Method of Moments for Mixture Models and Hidden Markov Models
TLDR
This work develops an efficient method of moments approach to parameter estimation for a broad class of high-dimensional mixture models with many components, including multi-view mixtures of Gaussians (such as mixture of axis-aligned Gaussian and hidden Markov models).
A Spectral Algorithm for Latent Tree Graphical Models
TLDR
This work proposes a fast, local-minimum-free spectral algorithm for learning latent variable models with arbitrary tree topologies, and shows that the joint distribution of the observed variables can be reconstructed from the marginals of triples of observed variables irrespective of the maximum degree of the tree.
Tensor Decompositions and Applications
This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order
Learning Topic Models -- Going beyond SVD
TLDR
This paper formally justifies Nonnegative Matrix Factorization (NMF) as a main tool in this context, which is an analog of SVD where all vectors are nonnegative, and gives the first polynomial-time algorithm for learning topic models without the above two limitations.
Efficiently learning mixtures of two Gaussians
TLDR
This work provides a polynomial-time algorithm for this problem for the case of two Gaussians in $n$ dimensions (even if they overlap), with provably minimal assumptions on theGaussians, and polynometric data requirements, and efficiently performs near-optimal clustering.
A Spectral Algorithm for Learning Hidden Markov Models
Tensor Methods in Statistics
This book provides a systematic development of tensor methods in statistics, beginning with the study of multivariate moments and cumulants. The effect on moment arrays and on cumulant arrays of
...
1
2
3
4
5
...