# Tensor decompositions for learning latent variable models

@article{Anandkumar2014TensorDF, title={Tensor decompositions for learning latent variable models}, author={Anima Anandkumar and Rong Ge and Daniel J. Hsu and Sham M. Kakade and Matus Telgarsky}, journal={ArXiv}, year={2014}, volume={abs/1210.7559} }

This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models--including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation--which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the…

## 942 Citations

Tensor Decompositions for Learning Latent Variable Models (A Survey for ALT)

- Computer Science, MathematicsALT
- 2015

This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models--including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation--which exploits a certain tensor structure in their low-order observable moments typically, of second- and third-order.

The Application of Symmetric Tensor Decomposition to Parameter Estimation of Latent Variable Models

- Computer Science, Mathematics
- 2019

This paper analyzes the application of two symmetric tensor decomposition methods to parameter estimation of certain latent variable models and examines the method of orthogonalization proposed by Anandkumar et al. and implemented through the tensor power method.

Nonparametric Estimation of Multi-View Latent Variable Models

- Computer ScienceICML
- 2014

A kernel method for learning multi-view latent variable models, allowing each mixture component to be nonparametric, and then the latent parameters are recovered using a robust tensor power method.

Smoothed analysis of tensor decompositions

- Computer ScienceSTOC
- 2014

This work introduces a smoothed analysis model for studying generative models and develops an efficient algorithm for tensor decomposition in the highly overcomplete case (rank polynomial in the dimension) and shows that tensor products of perturbed vectors are linearly independent in a robust sense.

Learning Negative Mixture Models by Tensor Decompositions

- Computer ScienceArXiv
- 2014

This work proposes a method to estimate the parameters of negative mixture models having a specific tensor structure in their low order observable moments, and introduces a generalization of the tensor power method for complex valued tensors, and establishes theoretical convergence guarantees.

CP Factor Model for Dynamic Tensors

- Computer Science
- 2021

A new high order projection estimator is proposed for such a factor model, utilizing the special structure and the idea of the higher order orthogonal iteration procedures commonly used in Tucker-type tensor factor model and general tensor CP decomposition procedures.

Discovery of Latent Factors in High-dimensional Data Using Tensor Methods

- Computer ScienceArXiv
- 2016

This is the first work that gives global convergence guarantees for the stochastic gradient descent on non-convex functions with exponentially many local minima and saddle points and large-scale deployment of spectral methods (matrix and tensor decomposition) carried out on CPU, GPU and Spark platforms.

Latent Dirichlet Allocation on Spark via Tensor Decomposition

- Computer Science
- 2015

This work proposes a framework to overcome the problem of unscalable and non-convex likelihood by taking the power of inverse method of moments and incorporates dimensionality reduction and thus is scalable to high-dimensional data.

Uniqueness of Tensor Decompositions with Applications to Polynomial Identifiability

- Mathematics, Computer ScienceCOLT
- 2014

It is proved that given a tensor whose decomposition satisfies a robust form of Kruskal's rank condition, it is possible to approximately recover the decomposition if the tensor is known up to a sufficiently small (inverse polynomial) error.

Tensor decomposition for learning Gaussian mixtures from moments

- Computer Science, MathematicsJ. Symb. Comput.
- 2022

## References

SHOWING 1-10 OF 103 REFERENCES

Smoothed analysis of tensor decompositions

- Computer ScienceSTOC
- 2014

This work introduces a smoothed analysis model for studying generative models and develops an efficient algorithm for tensor decomposition in the highly overcomplete case (rank polynomial in the dimension) and shows that tensor products of perturbed vectors are linearly independent in a robust sense.

A Spectral Algorithm for Latent Dirichlet Allocation

- Computer ScienceAlgorithmica
- 2014

This work provides a simple and efficient learning procedure that is guaranteed to recover the parameters for a wide class of multi-view models and topic models, including latent Dirichlet allocation (LDA).

Identifiability of parameters in latent structure models with many observed variables

- Mathematics, Computer Science
- 2009

A general approach for establishing identifiability utilizing algebraic arguments is demonstrated, which sheds light on the properties of finite mixtures of Bernoulli products, which have been used for decades despite being known to have nonidentifiable parameters.

A Method of Moments for Mixture Models and Hidden Markov Models

- Computer ScienceCOLT
- 2012

This work develops an efficient method of moments approach to parameter estimation for a broad class of high-dimensional mixture models with many components, including multi-view mixtures of Gaussians (such as mixture of axis-aligned Gaussian and hidden Markov models).

A Spectral Algorithm for Latent Tree Graphical Models

- Computer ScienceICML
- 2011

This work proposes a fast, local-minimum-free spectral algorithm for learning latent variable models with arbitrary tree topologies, and shows that the joint distribution of the observed variables can be reconstructed from the marginals of triples of observed variables irrespective of the maximum degree of the tree.

Tensor Decompositions and Applications

- Computer ScienceSIAM Rev.
- 2009

This survey provides an overview of higher-order tensor decompositions, their applications, and available software. A tensor is a multidimensional or $N$-way array. Decompositions of higher-order…

Learning Topic Models -- Going beyond SVD

- Computer Science2012 IEEE 53rd Annual Symposium on Foundations of Computer Science
- 2012

This paper formally justifies Nonnegative Matrix Factorization (NMF) as a main tool in this context, which is an analog of SVD where all vectors are nonnegative, and gives the first polynomial-time algorithm for learning topic models without the above two limitations.

Efficiently learning mixtures of two Gaussians

- Computer Science, MathematicsSTOC '10
- 2010

This work provides a polynomial-time algorithm for this problem for the case of two Gaussians in $n$ dimensions (even if they overlap), with provably minimal assumptions on theGaussians, and polynometric data requirements, and efficiently performs near-optimal clustering.

Tensor Methods in Statistics

- Mathematics
- 1987

This book provides a systematic development of tensor methods in statistics, beginning with the study of multivariate moments and cumulants. The effect on moment arrays and on cumulant arrays of…