Corpus ID: 233307503

Distill on the Go: Online knowledge distillation in self-supervised learning

@article{Bhat2021DistillOT,
  title={Distill on the Go: Online knowledge distillation in self-supervised learning},
  author={Prashant Bhat and E. Arani and Bahram Zonooz},
  journal={ArXiv},
  year={2021},
  volume={abs/2104.09866}
}
Self-supervised learning solves pretext prediction tasks that do not require annotations to learn feature representations. For vision tasks, pretext tasks such as predicting rotation, solving jigsaw are solely created from the input data. Yet, predicting this known information helps in learning representations useful for downstream tasks. However, recent works have shown that wider and deeper models benefit more from self-supervised learning than smaller models. To address the issue of self… Expand

Figures and Tables from this paper

References

SHOWING 1-10 OF 56 REFERENCES
Predicting What You Already Know Helps: Provable Self-Supervised Learning
Online Knowledge Distillation via Collaborative Learning
Knowledge Distillation by On-the-Fly Native Ensemble
Relational Knowledge Distillation
Knowledge Distillation Beyond Model Compression
Deep Mutual Learning
Similarity-Preserving Knowledge Distillation
A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning
Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty
...
1
2
3
4
5
...