Corpus ID: 220647258

Multi-label Contrastive Predictive Coding

@article{Song2020MultilabelCP,
  title={Multi-label Contrastive Predictive Coding},
  author={Jiaming Song and Stefano Ermon},
  journal={ArXiv},
  year={2020},
  volume={abs/2007.09852}
}
Variational mutual information (MI) estimators are widely used in unsupervised representation learning methods such as contrastive predictive coding (CPC). A lower bound on MI can be obtained from a multi-class classification problem, where a critic attempts to distinguish a positive sample drawn from the underlying joint distribution from $(m-1)$ negative samples drawn from a suitable proposal distribution. Using this approach, MI estimates are bounded above by $\log m$, and could thus… Expand

Figures and Tables from this paper

Hard Negative Mixing for Contrastive Learning
TLDR
It is argued that an important aspect of contrastive learning, i.e., the effect of hard negatives, has so far been neglected and proposed hard negative mixing strategies at the feature level, that can be computed on-the-fly with a minimal computational overhead. Expand
Multi-Label Contrastive Learning for Abstract Visual Reasoning
TLDR
Contrary to applications of contrastive learning methods reported in other domains, the state-of-the-art performance reported in the paper is achieved with no need for large batch sizes or strong data augmentation. Expand
PURSUhInT: In Search of Informative Hint Points Based on Layer Clustering for Knowledge Distillation
TLDR
The results show that hint points selected by the proposed algorithm results in superior compression performance with respect to state-of-the-art knowledge distillation algorithms on the same student models and datasets. Expand
Heterogeneous Contrastive Learning
TLDR
This paper proposes a unified heterogeneous learning framework, which combines both weighted unsupervised contrastive loss and weighted supervised Contrastive loss to model multiple types of heterogeneity. Expand
Decomposed Mutual Information Estimation for Contrastive Representation Learning
TLDR
It is shown that DEMI can capture a larger amount of MI than standard non-decomposed contrastive bounds in a synthetic setting, and learns better representations in a vision domain and for dialogue generation. Expand
Self-Contrastive Learning
This paper proposes a novel contrastive learning framework, coined as SelfContrastive (SelfCon) Learning, that self-contrasts within multiple outputs from the different levels of a network. WeExpand
A Low Rank Promoting Prior for Unsupervised Contrastive Learning
  • Yu Wang, Jingyang Lin, +4 authors Tao Mei
  • Computer Science
  • ArXiv
  • 2021
TLDR
A novel probabilistic graphical model is constructed that effectively incorporates the low rank promoting prior into the framework of contrastive learning, referred to as LORAC and clearly surpasses the state-of-the-art approaches on multiple benchmarks, including image classification, object detection, instance segmentation and keypoint detection. Expand
D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
TLDR
Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation and contrastive selfsupervised learning to improve representation quality is described. Expand
Integrating Auxiliary Information in Self-supervised Learning
TLDR
The Clustering InfoNCE (Cl-InfoNCE) objective is introduced that learns similar representations for augmented variants of data from the same cluster and dissimilar representations for data from different clusters and connects the goodness of the learned representations with the statistical relationships. Expand
Large-Margin Contrastive Learning with Distance Polarization Regularizer
TLDR
This paper proposes large-margin contrastive learning (LMCL) with distance polarization regularizer, motivated by the distribution characteristic of pairwise distances in metric learning, and proves a tighter error bound for LMCL. Expand
...
1
2
...

References

SHOWING 1-10 OF 57 REFERENCES
Striking the Right Balance With Uncertainty
TLDR
This paper demonstrates that the Bayesian uncertainty estimates directly correlate with the rarity of classes and the difficulty level of individual samples, and presents a novel framework for uncertainty based class imbalance learning that efficiently utilizes sample and class uncertainty information to learn robust features and more generalizable classifiers. Expand
Understanding the Limitations of Variational Mutual Information Estimators
TLDR
A new estimator is developed that focuses on variance reduction and theoretically shows that, under some conditions, estimators such as MINE exhibit variance that could grow exponentially with the true amount of underlying MI. Expand
Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
TLDR
It is shown that the ranking-based variant of NCE gives consistent parameter estimates under weaker assumptions than the classification-based method, which is closely related to negative sampling methods, now widely used in NLP. Expand
Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination
TLDR
This work forms this intuition as a non-parametric classification problem at the instance-level, and uses noise-contrastive estimation to tackle the computational challenges imposed by the large number of instance classes. Expand
Representation Learning with Contrastive Predictive Coding
TLDR
This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments. Expand
On Variational Bounds of Mutual Information
TLDR
This work introduces a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance and demonstrates the effectiveness of these new bounds for estimation and representation learning. Expand
Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
TLDR
A theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound is proposed that replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling. Expand
Class-Balanced Loss Based on Effective Number of Samples
TLDR
This work designs a re-weighting scheme that uses the effective number of samples for each class to re-balance the loss, thereby yielding a class-balanced loss and introduces a novel theoretical framework to measure data overlap by associating with each sample a small neighboring region rather than a single point. Expand
Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics
TLDR
The basic idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise and it is shown that the new method strikes a competitive trade-off in comparison to other estimation methods for unnormalized models. Expand
Deep Imbalanced Learning for Face Recognition and Attribute Prediction
TLDR
Cluster-based Large Margin Local Embedding (CLMLE), when combined with a simple k-nearest cluster algorithm, shows significant improvements in accuracy over existing methods on both face recognition and face attribute prediction tasks that exhibit imbalanced class distribution. Expand
...
1
2
3
4
5
...