Corpus ID: 225103395

Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

@article{Nguyen2020DoWA,
  title={Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth},
  author={Thao Nguyen and M. Raghu and Simon Kornblith},
  journal={ArXiv},
  year={2020},
  volume={abs/2010.15327}
}
  • Thao Nguyen, M. Raghu, Simon Kornblith
  • Published 2020
  • Computer Science
  • ArXiv
  • A key factor in the success of deep neural networks is the ability to scale models to improve performance by varying the architecture depth and width. This simple property of neural network design has resulted in highly effective architectures for a variety of tasks. Nevertheless, there is limited understanding of effects of depth and width on the learned representations. In this paper, we study this fundamental question. We begin by investigating how varying depth and width affects model… CONTINUE READING

    References

    SHOWING 1-10 OF 47 REFERENCES
    Wide Residual Networks
    • 2,663
    • Highly Influential
    • PDF
    The Expressive Power of Neural Networks: A View from the Width
    • 311
    • PDF
    Insights on representational similarity in neural networks with canonical correlation
    • 111
    • PDF
    FitNets: Hints for Thin Deep Nets
    • 1,268
    • PDF
    On the Expressive Power of Deep Neural Networks
    • 339
    • PDF
    EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
    • 1,354
    • PDF
    Deep Residual Learning for Image Recognition
    • 57,870
    • Highly Influential
    • PDF
    Gaussian Process Behaviour in Wide Deep Neural Networks
    • 191
    • PDF