Corpus ID: 211096972

Regularizing activations in neural networks via distribution matching with the Wasserstein metric

@article{Joo2020RegularizingAI,
  title={Regularizing activations in neural networks via distribution matching with the Wasserstein metric},
  author={Taejong Joo and Donggu Kang and Byunghoon Kim},
  journal={ArXiv},
  year={2020},
  volume={abs/2002.05366}
}
  • Taejong Joo, Donggu Kang, Byunghoon Kim
  • Published in ICLR 2020
  • Computer Science, Mathematics
  • Regularization and normalization have become indispensable components in training deep neural networks, resulting in faster training and improved generalization performance. We propose the projected error function regularization loss (PER) that encourages activations to follow the standard normal distribution. PER randomly projects activations onto one-dimensional space and computes the regularization loss in the projected space. PER is similar to the Pseudo-Huber loss in the projected space… CONTINUE READING

    Create an AI-powered research feed to stay up to date with new papers like this posted to ArXiv

    Citations

    Publications citing this paper.

    Topologically Densified Distributions

    VIEW 1 EXCERPT
    CITES METHODS

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 55 REFERENCES

    Regularizing by the Variance of the Activations' Sample-Variances

    VIEW 16 EXCERPTS
    HIGHLY INFLUENTIAL

    Deep Residual Learning for Image Recognition

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL

    Fixup Initialization: Residual Learning Without Normalization

    VIEW 8 EXCERPTS
    HIGHLY INFLUENTIAL

    Decorrelated Batch Normalization

    VIEW 9 EXCERPTS
    HIGHLY INFLUENTIAL

    Pointer Sentinel Mixture Models

    VIEW 6 EXCERPTS
    HIGHLY INFLUENTIAL