Contrastive Representation Learning: A Framework and Review

@article{LKhc2020ContrastiveRL,
  title={Contrastive Representation Learning: A Framework and Review},
  author={Ph{\'u}c H. L{\^e} Khắc and Graham Healy and Alan F. Smeaton},
  journal={IEEE Access},
  year={2020},
  volume={8},
  pages={193907-193934}
}
Contrastive Learning has recently received interest due to its success in self-supervised representation learning in the computer vision domain. However, the origins of Contrastive Learning date as far back as the 1990s and its development has spanned across many fields and domains including Metric Learning and natural language processing. In this paper, we provide a comprehensive literature review and we propose a general Contrastive Representation Learning framework that simplifies and… 
Negative Sampling for Contrastive Representation Learning: A Review
TLDR
This paper provides a systematic review of negative sampling techniques and summarizes the existing NS methods into four categories with pros and cons in each genre, and concludes with several open research questions as future directions.
Contrastive Continual Learning with Feature Propagation
TLDR
This paper proposes a general feature-propagation based contrastive continual learning method which is capable of handling multiple continual learning scenarios and aligns the current and previous representation spaces by means of feature propagation and contrastive representation learning to bridge the domain shifts among distinct tasks.
Investigating the Role of Negatives in Contrastive Representation Learning
TLDR
Theoretically, the existence of a collision-coverage trade-off is shown suggesting that the optimal number of negative examples should scale with the number of underlying concepts in the data, and empirical results find that the results broadly agree with theory and suggest future directions to better align theory and practice.
Sharp Learning Bounds for Contrastive Unsupervised Representation Learning
TLDR
By regarding the contrastive loss as a downstream loss estimator, the theory not only improves the existing learning bounds substantially but also explains why downstream classification empirically improves with larger negative samples—because the estimation variance of the downstream loss decays with largernegative samples.
Meta-Learning and Self-Supervised Pretraining for Real World Image Translation
TLDR
This work explores spatio-temporal structure in a recently introduced image-to-image translation problem in order to formulate a novel multi-task few-shot image generation benchmark and explore data augmentations in contrastive pre-training for image translation downstream tasks.
Learning Task-Relevant Representations with Selective Contrast for Reinforcement Learning in a Real-World Application
TLDR
The training of the reinforcement learning agent is presented, and the results show, that the choice of sampling distribution for negative samples is essential to allow task-relevant features to be represented in the presence of more prevalent, but irrelevant features.
Improving BERT Model Using Contrastive Learning for Biomedical Relation Extraction
TLDR
This work explores the method of employing contrastive learning to improve the text representation from the BERT model for relation extraction and investigates how large-scale data constructed from the external knowledge bases can enhance the generality of contrastive pre-training of BERT.
A Framework using Contrastive Learning for Classification with Noisy Labels
TLDR
An extensive empirical study is provided showing that a preliminary contrastive learning step brings a significant gain in performance when using different loss functions: non robust, robust, and early-learning regularized.
Revisiting Low-Resolution Images Retrieval with Attention Mechanism and Contrastive Learning
TLDR
This study considered a contrastive learning framework fine-tuned on features extracted from a pre-trained neural network encoder equipped with an attention mechanism to address the image retrieval task for low-resolution image retrieval.
Contrastive Coding for Active Learning under Class Distribution Mismatch
TLDR
CCAL is the first work related to AL for class distribution mismatch and extracts both semantic and distinctive features by contrastive learning and combines them in a query strategy to choose the most informative un-labeled samples with matched categories.
...
...

References

SHOWING 1-10 OF 130 REFERENCES
A Simple Framework for Contrastive Learning of Visual Representations
TLDR
It is shown that composition of data augmentations plays a critical role in defining effective predictive tasks, and introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning.
What Should Not Be Contrastive in Contrastive Learning
TLDR
This work introduces a contrastive learning framework which does not require prior knowledge of specific, task-dependent invariances, and learns to capture varying and invariant factors for visual representations by constructing separate embedding spaces, each of which is invariant to all but one augmentation.
What makes for good views for contrastive learning
TLDR
This paper uses empirical analysis to better understand the importance of view selection, and argues that the mutual information (MI) between views should be reduced while keeping task-relevant information intact, and devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI.
Representation Learning with Contrastive Predictive Coding
TLDR
This work proposes a universal unsupervised learning approach to extract useful representations from high-dimensional data, which it calls Contrastive Predictive Coding, and demonstrates that the approach is able to learn useful representations achieving strong performance on four distinct domains: speech, images, text and reinforcement learning in 3D environments.
Online Object Representations with Contrastive Learning
TLDR
A self-supervised approach for learning representations of objects from monocular videos is proposed and found that given a limited set of objects, object correspondences will naturally emerge when using contrastive learning without requiring explicit positive pairs.
Prototypical Contrastive Learning of Unsupervised Representations
TLDR
This paper introduces prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework and proposes ProtoNCE loss, a generalized version of the InfoN CE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes.
Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases
TLDR
This work demonstrates that approaches like MOCO and PIRL learn occlusion-invariant representations, but they fail to capture viewpoint and category instance invariance which are crucial components for object recognition, and proposes an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance.
Momentum Contrast for Unsupervised Visual Representation Learning
We present Momentum Contrast (MoCo) for unsupervised visual representation learning. From a perspective on contrastive learning as dictionary look-up, we build a dynamic dictionary with a queue and a
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
TLDR
This paper proposes an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons, and uses a swapped prediction mechanism where it predicts the cluster assignment of a view from the representation of another view.
Contrastive Learning of Structured World Models
TLDR
These experiments demonstrate that C-SWMs can overcome limitations of models based on pixel reconstruction and outperform typical representatives of this model class in highly structured environments, while learning interpretable object-based representations.
...
...