Corpus ID: 220042269

Towards Understanding Hierarchical Learning: Benefits of Neural Representations

@article{Chen2020TowardsUH,
  title={Towards Understanding Hierarchical Learning: Benefits of Neural Representations},
  author={Minshuo Chen and Yu Bai and J. Lee and Tuo Zhao and Huan Wang and Caiming Xiong and R. Socher},
  journal={ArXiv},
  year={2020},
  volume={abs/2006.13436}
}
Deep neural networks can empirically perform efficient hierarchical learning, in which the layers learn useful representations of the data. However, how they make use of the intermediate representations are not explained by recent theories that relate them to "shallow learners" such as kernels. In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks and can be advantageous over raw inputs. We consider a fixed, randomly initialized neural… Expand
Mathematical Models of Overparameterized Neural Networks
Deep Equals Shallow for ReLU Networks in Kernel Regimes
Spectral Leakage and Rethinking the Kernel Size in CNNs
Understanding Deflation Process in Over-parametrized Tensor Decomposition
  • Rong Ge, Yunwei Ren, Xiang Wang, Mo Zhou
  • Computer Science, Mathematics
  • ArXiv
  • 2021

References

SHOWING 1-10 OF 58 REFERENCES
A Convergence Theory for Deep Learning via Over-Parameterization
Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
On the importance of single directions for generalization
Efficient Approximation of Deep ReLU Networks for Functions on Low Dimensional Manifolds
Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
Limitations of Lazy Training of Two-layers Neural Networks
Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
...
1
2
3
4
5
...