Variational Neural Machine Translation with Normalizing Flows

@article{Setiawan2020VariationalNM,
  title={Variational Neural Machine Translation with Normalizing Flows},
  author={Hendra Setiawan and Matthias Sperber and Udhay Nallasamy and Matthias Paulik},
  journal={ArXiv},
  year={2020},
  volume={abs/2005.13978}
}
Variational Neural Machine Translation (VNMT) is an attractive framework for modeling the generation of target translations, conditioned not only on the source sentence but also on some latent random variables. The latent variable modeling may introduce useful statistical dependencies that can improve translation accuracy. Unfortunately, learning informative latent variables is non-trivial, as the latent space can be prohibitively large, and the latent codes are prone to be ignored by many… 

Tables from this paper

FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models

This work proposes adding the importance-sampled log marginal likelihood as a second term to the standard VAE objective to help when learning the prior, and demonstrates that the model, which is called FlowPrior, shows a substantial improvement in language modeling tasks compared to strong baselines.

Flow-Adapter Architecture for Unsupervised Machine Translation

In this work, we propose a flow-adapter architecture for unsupervised NMT. It leverages normalizing flows to explicitly model the distributions of sentence-level latent representations, which are

Constrained Density Matching and Modeling for Cross-lingual Alignment of Contextualized Representations

This work introduces supervised and unsupervised density-based approaches named Real-NVP and GAN-Real- NVP, driven by Normalizing Flow, to perform alignment, both dissecting the alignment of multilingual subspaces into density matching and density modeling, and complement these approaches with the validation criteria in order to guide the training process.

Survey of Low-Resource Machine Translation

A survey covering the state of the art in low-resource machine translation (MT) research is presented and a description of the techniques evaluated by researchers in several recent shared tasks inLow-resource MT is provided.

PortaSpeech: Portable and High-Quality Generative Text-to-Speech

PortaSpeech is proposed, a portable and high-quality generative text-to-speech model that outperforms other TTS models in both voice quality and prosody modeling in terms of subjective and objective evaluation metrics, and shows only a slight performance degradation when reducing the model parameters to 6.7M.

PortaSpeech: Portable and High-Quality Generative Text-to-Speech

PortaSpeech is proposed, a portable and high-quality generative text-to-speech model that outperforms other TTS models in both voice quality and prosody modeling in terms of subjective and objective evaluation metrics, and shows only a slight performance degradation when reducing the model parameters to 6.7M.

PortaSpeech: Portable and High-Quality Generative Text-to-Speech

PortaSpeech is proposed, a portable and high-quality generative text-to-speech model that outperforms other TTS models in both voice quality and prosody modeling in terms of subjective and objective evaluation metrics, and shows only a slight performance degradation when reducing the model parameters to 6.7M.

References

SHOWING 1-10 OF 25 REFERENCES

Conditional Variational Autoencoder for Neural Machine Translation

This is the first reported conditional variational model for text that meaningfully utilizes the latent variable without weakening the translation model, and is extended with a co-attention mechanism motivated by Parikh et al. in the inference network.

Auto-Encoding Variational Neural Machine Translation

We present a deep generative model of bilingual sentence pairs for machine translation. The model generates source and target sentences jointly from a shared latent representation and is

Variational Neural Machine Translation

This paper builds a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization technique to estimate the variational lower bound, and shows that the proposed variational neural machine translation achieves significant improvements over the vanilla neural machinetranslation baselines.

Variational Recurrent Neural Machine Translation

A novel variational recurrent neural machine translation model that introduces a series of latent random variables to model the translation procedure of a sentence in a generative way, instead of a single latent variable.

Generative Neural Machine Translation

Generative Neural Machine Translation is introduced, a latent variable architecture which is designed to model the semantics of the source and target sentences and achieves competitive BLEU scores on pure translation tasks, and is superior when there are missing words in the source sentence.

Investigating the impact of normalizing flows on latent variable machine translation

The results suggest that normalizing flows can improve translation quality in some scenarios, and require certain modelling assumptions to achieve such improvements.

Improved Variational Neural Machine Translation by Promoting Mutual Information

This work extends the conditional variational autoencoder with two new ingredients: first, it proposes a modified evidence lower bound (ELBO) objective which explicitly promotes mutual information; second, it regularize the probabilities of the decoder by mixing an auxiliary factorized distribution which is directly predicted by the latent variables.

A Convolutional Encoder Model for Neural Machine Translation

A faster and simpler architecture based on a succession of convolutional layers that allows to encode the source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies is presented.

Generating Sentences from a Continuous Space

This work introduces and study an RNN-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences that allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features.

Understanding Back-Translation at Scale

This work broadens the understanding of back-translation and investigates a number of methods to generate synthetic source sentences, finding that in all but resource poor settings back-translations obtained via sampling or noised beam outputs are most effective.