• Corpus ID: 232307426

Self-supervised Representation Learning with Relative Predictive Coding

  title={Self-supervised Representation Learning with Relative Predictive Coding},
  author={Yao-Hung Hubert Tsai and Martin Q. Ma and Muqiao Yang and Han Zhao and Louis-Philippe Morency and Ruslan Salakhutdinov},
This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity, and downstream task performance. The key to the success of RPC is two-fold. First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance. Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in… 

Figures and Tables from this paper

f -Mutual Information Contrastive Learning

It is theoretically prove that the proposed f -Mutual Information Contrastive Learning framework (f -MICL) naturally attains the alignment for positive pairs and the uniformity for data representations, the two main factors for the success of contrastive learning.

TiCo: Transformation Invariance and Covariance Contrast for Self-Supervised Visual Representation Learning

By connecting the contrastive and redundancy-reduction methods together, TiCo gives us new insights into how joint embedding methods work.

Decoupled Contrastive Learning

This work identifies a noticeable negative-positive-coupling effect in the widely used InfoNCE loss, leading to unsuitable learning efficiency concerning the batch size, and proposes decoupled contrastive learning (DCL) loss, which removes the positive term from the denominator and significantly improves the learning efficiency.

Conditional Contrastive Learning: Removing Undesirable Information in Self-Supervised Representations

Conditional contrastive learning to remove undesirable information in self-supervised representations is developed and its computationally efficient variant, Weak-Conditional InfoNCE (WeaC-InfoNCE), is introduced for conditional Contrastive learning.


  • Computer Science
  • 2021
This work identifies a noticeable negative-positive-coupling (NPC) effect in the widely used cross-entropy (InfoNCE) loss, leading to unsuitable learning efficiency with respect to the batch size, and reaches a decoupled contrastive learning (DCL) objective function, significantly improving SSL efficiency.

A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning

Compared to the prior state-of-the-art SSL methods, Barlow Twins demonstrates two main properties: its algorithm requires no explicit construction of negative sample pairs, and is not sensitive to large training batch sizes.

Learning Weakly-Supervised Contrastive Representations

It is argued that a form of the valuable information provided by the auxiliary information is its implied data clustering information, and a two-stage weakly-supervised contrastive learning approach is presented, which performs the best in most cases, when comparing with other baseline representation learning methods that also leverage auxiliary data information.

Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning

Experiments show that the proposed CARE framework improves CNN encoder backbones to the state-of-the-art performance on several standard visual recognition benchmarks, including image classification, object detection, and semantic segmentation.

Integrating Auxiliary Information in Self-supervised Learning

The Clustering InfoNCE (Cl-InfoNCE) objective is introduced that learns similar representations for augmented variants of data from the same cluster and dissimilar representations for data from different clusters and connects the goodness of the learned representations with the statistical relationships.

Conditional Contrastive Learning with Kernel

This paper presents Conditional Contrastive Learning with Kernel (CCL-K) that converts existing conditional contrastive objectives into alternative forms that mitigate the insufficient data problem, and conducts experiments using weakly supervised, fair, and hard negatives contrastive learning, showing CCL-K outperforms state-of-the-art baselines.



Representation Learning with Contrastive Predictive Coding

This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.

A Simple Framework for Contrastive Learning of Visual Representations

It is shown that composition of data augmentations plays a critical role in defining effective predictive tasks, and introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning.

Demystifying Self-Supervised Learning: An Information-Theoretical Framework

It is demonstrated that self-supervisedly learned representation can extract task-relevant and discard task-irrelevant information under the assumption that only the shared information between the input and self- supervised signals contributes to downstream tasks.

Generative Pretraining From Pixels

This work trains a sequence Transformer to auto-regressively predict pixels, without incorporating knowledge of the 2D input structure, and finds that a GPT-2 scale model learns strong image representations as measured by linear probing, fine-tuning, and low-data classification.

Learning deep representations by mutual information estimation and maximization

It is shown that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks and is an important step towards flexible formulations of representation learning objectives for specific end-goals.

What makes for good views for contrastive learning

This paper uses empirical analysis to better understand the importance of view selection, and argues that the mutual information (MI) between views should be reduced while keeping task-relevant information intact, and devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI.

Momentum Contrast for Unsupervised Visual Representation Learning

We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a

Contrastive learning, multi-view redundancy, and linear models

This work provides a theoretical analysis of contrastive learning in the multi-view setting, where two views of each datum are available, and finds that linear functions of the learned representations are nearly optimal on downstream prediction tasks whenever the two views provide redundant information about the label.

Debiased Contrastive Learning

A debiased contrastive objective is developed that corrects for the sampling of same-label datapoints, even without knowledge of the true labels, and consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks.

Wasserstein Dependency Measure for Representation Learning

It is empirically demonstrated that mutual information-based representation learning approaches do fail to learn complete representations on a number of designed and real-world tasks, and a practical approximation to this theoretically motivated solution, constructed using Lipschitz constraint techniques from the GAN literature, achieves substantially improved results on tasks where incomplete representations are a major challenge.