• Corpus ID: 238583584

Discriminative Multimodal Learning via Conditional Priors in Generative Models

@article{Mancisidor2021DiscriminativeML,
  title={Discriminative Multimodal Learning via Conditional Priors in Generative Models},
  author={Rogelio Andrade Mancisidor and Michael C. Kampffmeyer and Kjersti Aas and Robert Jenssen},
  journal={ArXiv},
  year={2021},
  volume={abs/2110.04616}
}
Deep generative models with latent variables have been used lately to learn joint representations and generative processes from multi-modal data. These two learning mechanisms can, however, conflict with each other and representations can fail to embed information on the data modalities. This research studies the realistic scenario in which all modalities and class labels are available for model training, but where some modalities and labels required for downstream tasks are missing. We show, in… 

Figures and Tables from this paper

References

SHOWING 1-10 OF 45 REFERENCES

Multimodal Generative Models for Scalable Weakly-Supervised Learning

A multimodal variational autoencoder that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem and shares parameters to efficiently learn under any combination of missing modalities, thereby enabling weakly-supervised learning.

Joint Multimodal Learning with Deep Generative Models

The proposed joint multimodal variational autoencoder (JMVAE), in which all modalities are independently conditioned on joint representation, models a joint distribution of modalities and can generate and reconstruct them more properly than conventional VAEs.

Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models

This work proposes a mixture-of-experts multi-modal variational autoencoder (MMVAE) for learning of generative models on different sets of modalities, including a challenging image language dataset, and demonstrates its ability to satisfy all four criteria, both qualitatively and quantitatively.

Improving Generative Modelling in VAEs Using Multimodal Prior

In this paper we propose a conditional generative modelling (CGM) approach for unsupervised disentangled representation learning using variational autoencoder (VAE). CGM employs a

Discriminative multi-modal deep generative models

Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence

A novel, efficient objective function that utilizes the Jensen-Shannon divergence for multiple distributions and simultaneously approximates the unimodal and joint multimodal posteriors directly via a dynamic prior is proposed.

Learning Structured Output Representation using Deep Conditional Generative Models

A deep conditional generative model for structured output prediction using Gaussian latent variables is developed, trained efficiently in the framework of stochastic gradient variational Bayes, and allows for fast prediction using Stochastic feed-forward inference.

Multimodal learning with deep Boltzmann machines

A Deep Boltzmann Machine is proposed for learning a generative model of multimodal data and it is shown that the model can be used to create fused representations by combining features across modalities, which are useful for classification and information retrieval.

Multimodal Adversarially Learned Inference with Factorized Discriminators

This work proposes a novel approach to generative modeling of multimodal data based on generative adversarial networks by taking advantage of contrastive learning through factorizing the discriminator, which outperforms the-state-ofthe-art methods on a variety of metrics.

Avoiding Latent Variable Collapse With Generative Skip Models

Compared to existing VAE architectures, it is shown that generative skip models maintain similar predictive performance but lead to less collapse and provide more meaningful representations of the data.