A Spectral Algorithm for Latent Dirichlet Allocation

@article{Anandkumar2014ASA,
  title={A Spectral Algorithm for Latent Dirichlet Allocation},
  author={Anima Anandkumar and Dean P. Foster and Daniel J. Hsu and S. Kakade and Yi-Kai Liu},
  journal={Algorithmica},
  year={2014},
  volume={72},
  pages={193-214}
}
Topic modeling is a generalization of clustering that posits that observations (words in a document) are generated by multiple latent factors (topics), as opposed to just one. The increased representational power comes at the cost of a more challenging unsupervised learning problem for estimating the topic-word distributions when only words are observed, and the topics are hidden. This work provides a simple and efficient learning procedure that is guaranteed to recover the parameters for a… Expand
Fast and Sample-Efficient Learning for Latent Dirichlet Allocation
TLDR
If there are k topics and typically α0 topics per document, the authors can learn the topic-word distributions for LDA in polynomial time from only roughly k · α0 documents, and this bound on the sample complexity is tight up to poly-logarithmic factors. Expand
Spectral Methods for Correlated Topic Models
TLDR
It is proved that this flexible topic model class can be learned via spectral methods using only moments up to the third order, with (low order) polynomial sample and computational complexity. Expand
Online Bayesian Moment Matching for Topic Modeling with Unknown Number of Topics
TLDR
Two new models that extend LDA in a simple and intuitive fashion by directly expressing a distribution over the number of topics are proposed and a new online Bayesian moment matching technique to learn the parameters and theNumber of topics of those models based on streaming data is proposed. Expand
Spectral Learning for Supervised Topic Models
TLDR
The results on a large-scale review rating dataset demonstrate that the single-phase spectral algorithm alone gets comparable or even better performance than state-of-the-art methods, while previous work on spectral methods has rarely reported such promising performance. Expand
Likelihood estimation of sparse topic distributions in topic models and its applications to Wasserstein document distance calculations
This paper studies the estimation of high-dimensional, discrete, possibly sparse, mixture models in the context topic models. The data consists of observed multinomial counts of p words across nExpand
Beyond LDA: A Unified Framework for Learning Latent Normalized Infinitely Divisible Topic Models through Spectral Methods
TLDR
This paper overcome the limitation of LDA to incorporate arbitrary topic correlations, by assuming that the hidden topic proportions are drawn from a flexible class of Normalized Infinitely Divisible (NID) distributions. Expand
A new method of moments for latent variable models
TLDR
An algorithm for the unsupervised learning of latent variable models based on the method of moments is presented, and the proposed tensor decomposition technique achieves the learning accuracy of the state-of-the-art method with significantly smaller running times. Expand
Scalable Moment-Based Inference for Latent Dirichlet Allocation
TLDR
This work proposes a speedup technique by leveraging the special structure of the tensors that improves over the state-of-the-art inference algorithm by one to three orders of magnitude, while preserving equal inference ability. Expand
Prior-aware Dual Decomposition: Document-specific Topic Inference for Spectral Topic Models
TLDR
Experiments show that PADD successfully leverages topic correlations as a prior, notably outperforming TLI and learning quality topic compositions comparable to Gibbs sampling on various data. Expand
A provable SVD-based algorithm for learning topics in dominant admixture corpus
TLDR
Under a more realistic assumption, a singular value decomposition (SVD) based algorithm with a crucial pre-processing step of thresholding, can provably recover the topics from a collection of documents drawn from Dominant admixtures. Expand
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 85 REFERENCES
Learning Topic Models -- Going beyond SVD
TLDR
This paper formally justifies Nonnegative Matrix Factorization (NMF) as a main tool in this context, which is an analog of SVD where all vectors are nonnegative, and gives the first polynomial-time algorithm for learning topic models without the above two limitations. Expand
Latent Dirichlet Allocation
We propose a generative model for text and other collections of discrete data that generalizes or improves on several previous models including naive Bayes/unigram, mixture of unigrams [6], andExpand
Tensor decompositions for learning latent variable models
TLDR
A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices, and implies a robust and computationally tractable estimation approach for several popular latent variable models. Expand
Online Learning for Latent Dirichlet Allocation
TLDR
An online variational Bayes (VB) algorithm for Latent Dirichlet Allocation (LDA) based on online stochastic optimization with a natural gradient step is developed, which shows converges to a local optimum of the VB objective function. Expand
GaP: a factor model for discrete data
TLDR
The GaP model projects documents and terms into a low-dimensional space of "themes," and models texts as "passages" of terms on the same theme, and gives very accurate results as a probabilistic model (measured via perplexity) and as a retrieval model. Expand
Contrastive Learning Using Spectral Methods
TLDR
This paper formalizes this notion of contrastive learning for mixture models, and develops spectral algorithms for inferring mixture components specific to a foreground data set when contrasted with a background data set. Expand
Multi-view clustering via canonical correlation analysis
TLDR
Under the assumption that the views are un-correlated given the cluster label, it is shown that the separation conditions required for the algorithm to be successful are significantly weaker than prior results in the literature. Expand
On Spectral Learning of Mixtures of Distributions
TLDR
It is proved that a very simple algorithm, namely spectral projection followed by single-linkage clustering, properly classifies every point in the sample, and there are many Gaussian mixtures such that each pair of means is separated, yet upon spectral projection the mixture collapses completely. Expand
Spectral Methods for Learning Multivariate Latent Tree Structure
This work considers the problem of learning the structure of multivariate linear tree models, which include a variety of directed tree graphical models with continuous, discrete, and mixed latentExpand
A Spectral Algorithm for Latent Dirichlet Allocation
Topic modeling is a generalization of clustering that posits that observations (words in a document) are generated by multiple latent factors (topics), as opposed to just one. The increased represe...
...
1
2
3
4
5
...