Learning by Sorting: Self-supervised Learning with Group Ordering Constraints

@article{Shvetsova2023LearningBS,
  title={Learning by Sorting: Self-supervised Learning with Group Ordering Constraints},
  author={Nina Shvetsova and Felix Petersen and Anna Kukleva and Bernt Schiele and Hilde Kuehne},
  journal={ArXiv},
  year={2023},
  volume={abs/2301.02009}
}
Contrastive learning has become a prominent ingredi-ent in learning representations from unlabeled data. However, existing methods primarily consider pairwise relations. This paper proposes a new approach towards self-supervised contrastive learning based on Group Ordering Constraints (GroCo). The GroCo loss leverages the idea of comparing groups of positive and negative images instead of pairs of images. Building on the recent success of differentiable sorting algorithms, group ordering… 
[PDF] Semantic Reader

References

SHOWING 1-10 OF 60 REFERENCES

Whitening for Self-Supervised Representation Learning

This paper proposes a different direction and a new loss function for self-supervised learning which is based on the whitening of the latent-space features and empirically shows that this loss accelerates self- supervised training and the learned representations are much more effective for downstream tasks than previously published work.

Unsupervised Learning of Visual Features by Contrasting Cluster Assignments

This paper proposes an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons, and uses a swapped prediction mechanism where it predicts the cluster assignment of a view from the representation of another view.

With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations

This work finds that using the nearest-neighbor as positive in contrastive losses improves performance significantly on ImageNet classification using ResNet-50 under the linear evaluation protocol, and demonstrates empirically that the method is less reliant on complex data augmentations.

Hard negative examples are hard, but useful

This paper characterize the space of triplets and derive why hard negatives make triplet loss training fail, and offers a simple fix to the loss function and shows that, with this fix, optimizing with hard negative examples becomes feasible.

What makes for good views for contrastive learning

This paper uses empirical analysis to better understand the importance of view selection, and argues that the mutual information (MI) between views should be reduced while keeping task-relevant information intact, and devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI.

Boosting Contrastive Self-Supervised Learning with False Negative Cancellation

This paper proposes novel approaches to identify false negatives, as well as two strategies to mitigate their effect, i.e. false negative elimination and attraction, while systematically performing rigorous evaluations to study this problem in detail.

Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision

This work proposes differentiable sorting networks by relaxing their pairwise conditional swap operations and proposes mapping activations to regions with moderate gradients to address the problems of vanishing gradients and extensive blurring that arise with larger numbers of layers.

Big Self-Supervised Models are Strong Semi-Supervised Learners

The proposed semi-supervised learning algorithm can be summarized in three steps: unsupervised pretraining of a big ResNet model using SimCLRv2 (a modification of SimCLRs), supervised fine-tuning on a few labeled examples, and distillation with unlabeled examples for refining and transferring the task-specific knowledge.

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning

This work introduces Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning that performs on par or better than the current state of the art on both transfer and semi- supervised benchmarks.

How Well Do Self-Supervised Models Transfer?

It is shown that on most tasks the best self-supervised models outperform supervision, confirming the recently observed trend in the literature and finding ImageNet Top-1 accuracy to be highly correlated with transfer to many-shot recognition, but increasingly less so for few-shot, object detection and dense prediction.
...