Corpus ID: 60440652

Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher

@article{Mirzadeh2019ImprovedKD,
  title={Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher},
  author={Seyed Iman Mirzadeh and Mehrdad Farajtabar and Ang Li and Hassan Ghasemzadeh},
  journal={ArXiv},
  year={2019},
  volume={abs/1902.03393}
}
  • Seyed Iman Mirzadeh, Mehrdad Farajtabar, +1 author Hassan Ghasemzadeh
  • Published 2019
  • Computer Science, Mathematics
  • ArXiv
  • Despite the fact that deep neural networks are powerful models and achieve appealing results on many tasks, they are too gigantic to be deployed on edge devices like smart-phones or embedded sensor nodes. There has been efforts to compress these networks, and a popular method is knowledge distillation, where a large (a.k.a. teacher) pre-trained network is used to train a smaller (a.k.a. student) network. However, in this paper, we show that the student network performance degrades when the gap… CONTINUE READING

    Citations

    Publications citing this paper.
    SHOWING 1-10 OF 63 CITATIONS

    Teacher-Class Network: A Neural Network Compression Mechanism

    VIEW 4 EXCERPTS
    CITES BACKGROUND & RESULTS
    HIGHLY INFLUENCED

    Residual Knowledge Distillation

    VIEW 2 EXCERPTS
    CITES METHODS

    An Embarrassingly Simple Approach for Knowledge Distillation

    VIEW 1 EXCERPT
    CITES BACKGROUND

    Role-Wise Data Augmentation for Knowledge Distillation

    VIEW 2 EXCERPTS
    CITES RESULTS & BACKGROUND

    Knowledge Distillation from Internal Representations

    VIEW 1 EXCERPT
    CITES BACKGROUND

    Knowledge Distillation via Route Constrained Optimization

    VIEW 2 EXCERPTS
    CITES BACKGROUND & METHODS

    Ensemble Knowledge Distillation for Learning Improved and Efficient Networks

    VIEW 4 EXCERPTS
    CITES METHODS
    HIGHLY INFLUENCED

    FILTER CITATIONS BY YEAR

    2018
    2020

    CITATION STATISTICS

    • 12 Highly Influenced Citations

    • Averaged 21 Citations per year from 2018 through 2020

    • 65% Increase in citations per year in 2020 over 2019

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 33 REFERENCES

    FitNets: Hints for Thin Deep Nets

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning

    VIEW 3 EXCERPTS

    Distilling the Knowledge in a Neural Network

    VIEW 23 EXCERPTS
    HIGHLY INFLUENTIAL