• Corpus ID: 3999202

Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC

  title={Deep Latent Dirichlet Allocation with Topic-Layer-Adaptive Stochastic Gradient Riemannian MCMC},
  author={Yulai Cong and Bo Chen and Hongwei Liu and Mingyuan Zhou},
It is challenging to develop stochastic gradient based scalable inference for deep discrete latent variable models (LVMs), due to the difficulties in not only computing the gradients, but also adapting the step sizes to different latent factors and hidden layers. For the Poisson gamma belief network (PGBN), a recently proposed deep discrete LVM, we derive an alternative representation that is referred to as deep latent Dirichlet allocation (DLDA). Exploiting data augmentation and… 
Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network
Seetooth factorial topic embedding guided GBN is proposed, a deep generative model of documents that captures the dependencies and semantic similarities between the topics in the embedding space and outperforms other neural topic models on extracting deeper interpretable topics and deriving better document representations.
Max-Margin Deep Diverse Latent Dirichlet Allocation With Continual Learning
This article proposes deep diverse latent Dirichlet allocation (DDLDA), a deep hierarchical topic model that can yield more meaningful semantic topics with less common and meaningless words by introducing shared topics and develops a variational inference network for DDLDA.
Decoupling Sparsity and Smoothness in the Dirichlet Variational Autoencoder Topic Model
This work rewrite the Dirichlet parameter vector into a product of a sparse binary vector and a smoothness vector, leading to a model that features both a competitive topic coherence and a high log-likelihood.
WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling
To train an inference network jointly with a deep generative topic model, making it both scalable to big corpora and fast in out-of-sample prediction, we develop Weibull hybrid autoencoding inference
Multimodal Weibull Variational Autoencoder for Jointly Modeling Image-Text Data.
A novel multimodal Poisson gamma belief network (mPGBN) is developed that tightly couples the observations of different modalities via imposing sparse connections between their modality-specific hidden layers, resulting in a novel Weibull variational autoencoder (MWVAE), which is fast in out-of-sample prediction and can handle large-scale multimodAL datasets.
Dirichlet belief networks for topic structure learning
A new multi-layer generative process on word distributions of topics, where each layer consists of a set of topics and each topic is drawn from a mixture of the topics of the layer above, which is able to discover interpretable topic hierarchies.
TopicNet: Semantic Graph-Guided Topic Discovery
This work introduces TopicNet, a deep hierarchical topic model that can inject prior structural knowledge as an inductive bias to guide the learning of the topic hierarchy.
Deep Dynamic Poisson Factorization Model
A new model, named as deep dynamic poisson factorization model, is proposed in this paper for analyzing sequential count vectors, based on the Poisson Factor Analysis method, and results show good predicting and fitting performance with interpretable latent structure.
Deep Poisson gamma dynamical systems
We develop deep Poisson-gamma dynamical systems (DPGDS) to model sequentially observed multivariate count data, improving previously proposed models by not only mining deep hierarchical latent


Neural Variational Inference and Learning in Belief Networks
This work proposes a fast non-iterative approximate inference method that uses a feedforward network to implement efficient exact sampling from the variational posterior and shows that it outperforms the wake-sleep algorithm on MNIST and achieves state-of-the-art results on the Reuters RCV1 document dataset.
Augmentable Gamma Belief Networks
An augmentable gamma belief network (GBN) that factorizes each of its hidden layers into the product of a sparse connection weight matrix and the nonnegative real hidden units of the next layer to infer multilayer deep representations of high-dimensional discrete and non negative real vectors.
Auto-Encoding Variational Bayes
A stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case is introduced.
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
This work proposes combining adaptive preconditioners with Stochastic Gradient Langevin Dynamics, and gives theoretical properties on asymptotic convergence and predictive risk, and empirical results for Logistic Regression, Feedforward Neural Nets, and Convolutional Neural Nets demonstrate that the preconditionsed SGLD method gives state-of-the-art performance.
Online Learning for Latent Dirichlet Allocation
An online variational Bayes (VB) algorithm for Latent Dirichlet Allocation (LDA) based on online stochastic optimization with a natural gradient step is developed, which shows converges to a local optimum of the VB objective function.
Neural Variational Inference for Text Processing
This paper introduces a generic variational inference framework for generative and conditional models of text, and constructs an inference network conditioned on the discrete text input to provide the variational distribution.
Learning Sigmoid Belief Networks via Monte Carlo Expectation Maximization
This work proposes using an online Monte Carlo expectationmaximization (MCEM) algorithm to learn the maximum a posteriori (MAP) estimator of the generative model or optimize the variational lower bound of a recognition network.
Pachinko allocation: DAG-structured mixture models of topic correlations
Improved performance of PAM is shown in document classification, likelihood of held-out data, the ability to support finer-grained topics, and topical keyword coherence.
Revisiting Natural Gradient for Deep Networks
It is described how one can use unlabeled data to improve the generalization error obtained by natural gradient and empirically evaluate the robustness of the algorithm to the ordering of the training set compared to stochastic gradient descent.
Replicated Softmax: an Undirected Topic Model
We introduce a two-layer undirected graphical model, called a "Replicated Softmax", that can be used to model and automatically extract low-dimensional latent semantic representations from a large