Corpus ID: 211252521

Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization

@article{Huesmann2020ExploitingTF,
  title={Exploiting the Full Capacity of Deep Neural Networks while Avoiding Overfitting by Targeted Sparsity Regularization},
  author={Karim Huesmann and Soeren Klemm and Lars Linsen and Benjamin Risse},
  journal={ArXiv},
  year={2020},
  volume={abs/2002.09237}
}
  • Karim Huesmann, Soeren Klemm, +1 author Benjamin Risse
  • Published 2020
  • Computer Science, Mathematics
  • ArXiv
  • Overfitting is one of the most common problems when training deep neural networks on comparatively small datasets. Here, we demonstrate that neural network activation sparsity is a reliable indicator for overfitting which we utilize to propose novel targeted sparsity visualization and regularization strategies. Based on these strategies we are able to understand and counteract overfitting caused by activation sparsity and filter correlation in a targeted layer-by-layer manner. Our results… CONTINUE READING

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 40 REFERENCES
    Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
    • 19,200
    • PDF
    Understanding deep learning requires rethinking generalization
    • 2,101
    • PDF
    Self-Normalizing Neural Networks
    • 846
    • PDF
    U-Net: Convolutional Networks for Biomedical Image Segmentation
    • 16,098
    • PDF
    Visualizing and Understanding Convolutional Networks
    • 8,439
    • PDF
    Reducing Overfitting in Deep Networks by Decorrelating Representations
    • 191
    • PDF
    Pruning Convolutional Neural Networks for Resource Efficient Inference
    • 526
    On Implicit Filter Level Sparsity in Convolutional Neural Networks
    • 12
    • PDF