Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence
- Alexander Miserlis Hoyle, Pranav Goel, Denis Peskov, Andrew Hian-Cheong, Jordan L. Boyd-Graber, P. Resnik
- Computer ScienceNeural Information Processing Systems
- 5 July 2021
A meta-analysis of topic modeling literature reveals a substantial standardization gap in au- tomated topic modeling benchmarks and compares automated coherence with the two most widely accepted human judgment tasks: topic rating and word intrusion.
Improving Neural Topic Models Using Knowledge Distillation
- Alexander Miserlis Hoyle, Pranav Goel, P. Resnik
- Computer ScienceConference on Empirical Methods in Natural…
- 5 October 2020
This work uses knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers to improve topic quality, and shows that the adaptable framework not only improves performance in the aggregate over all estimated topics, but also in head-to-head comparisons of aligned topics.
Unsupervised Discovery of Gendered Language through Latent-Variable Modeling
- Alexander Miserlis Hoyle, Lawrence Wolf-Sonkin, H. Wallach, Isabelle Augenstein, Ryan Cotterell
- SociologyAnnual Meeting of the Association for…
- 11 June 2019
A generative latent-variable model is introduced that jointly represents adjective (or verb) choice, with its sentiment, given the natural gender of a head (or dependent) noun.
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
- Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, Jordan L. Boyd-Graber
- Computer ScienceAnnual Meeting of the Association for…
- 2021
This work creates a Bayesian leaderboard model where latent subject skill and latent item difficulty predict correct responses and analyzes the ranking reliability of leaderboards to advocate a re-imagining.
Promoting Graph Awareness in Linearized Graph-to-Text Generation
- Alexander Miserlis Hoyle, Ana Marasović, Noah A. Smith
- Computer ScienceFindings
- 31 December 2020
This work uses graph-denoising objectives implemented in a multi-task text-to-text framework and finds that these denoising scaffolds lead to substantial improvements in downstream generation in low-resource settings.
Combining Sentiment Lexica with a Multi-View Variational Autoencoder
- Alexander Miserlis Hoyle, Lawrence Wolf-Sonkin, H. Wallach, Ryan Cotterell, Isabelle Augenstein
- Computer ScienceNorth American Chapter of the Association for…
- 5 April 2019
A generative model of sentiment lexica is introduced to combine disparate scales into a common latent representation and is realized with a novel multi-view variational autoencoder (VAE), called SentiVAE.
Are Neural Topic Models Broken?
- Alexander Miserlis Hoyle, Pranav Goel, Rupak Sarkar, P. Resnik
- Computer ScienceArXiv
- 28 October 2022
It is demonstrated that a straightforward ensembling method can reli-ably outperform the members of the ensemble and that neural topic models fare worse in both respects compared to an established classical method.