Corpus ID: 219559263

Knowledge Distillation: A Survey

@article{Gou2020KnowledgeDA,
  title={Knowledge Distillation: A Survey},
  author={Jianping Gou and Baosheng Yu and Stephen J. Maybank and Dacheng Tao},
  journal={ArXiv},
  year={2020},
  volume={abs/2006.05525}
}
  • Jianping Gou, Baosheng Yu, +1 author Dacheng Tao
  • Published 2020
  • Computer Science, Mathematics
  • ArXiv
  • In recent years, deep neural networks have been very successful in the fields of both industry and academia, especially for the applications of visual recognition and neural language processing. The great success of deep learning mainly owes to its great scalabilities to both large-scale data samples and billions of model parameters. However, it also poses a great challenge for the deployment of these cumbersome deep models on devices with limited resources, e.g., mobile phones and embedded… CONTINUE READING

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 289 REFERENCES

    Datafree knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535

    • R. G. Lopes, S. Fenu, T. Starner
    • 2017
    VIEW 7 EXCERPTS
    HIGHLY INFLUENTIAL

    Learning Lightweight Lane Detection CNNs by Self Attention Distillation

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    2019b). Learn spelling from teachers: transferring knowledge from language models to sequence-tosequence speech recognition. In: Interspeech

    • Y. Bai, J. Yi, J. Tao, Z. Tian, Z. Wen
    • 2019
    VIEW 5 EXCERPTS
    HIGHLY INFLUENTIAL

    A two-teacher tramework for knowledge distillation

    • X. Chen, J. Su, J. Zhang
    • ISNN
    • 2019
    VIEW 11 EXCERPTS
    HIGHLY INFLUENTIAL

    Attention-Guided Answer Distillation for Machine Reading Comprehension

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL