Corpus ID: 227229101

Feature Learning in Infinite-Width Neural Networks

@article{Yang2020FeatureLI,
  title={Feature Learning in Infinite-Width Neural Networks},
  author={Greg Yang and Edward J. Hu},
  journal={ArXiv},
  year={2020},
  volume={abs/2011.14522}
}

Greg Yang, Edward J. Hu

Published 2020

Computer Science, Physics

ArXiv

As its width tends to infinity, a deep neural network's behavior under gradient descent can become simplified and predictable (e.g. given by the Neural Tangent Kernel (NTK)), if it is parametrized appropriately (e.g. the NTK parametrization). However, we show that the standard and NTK parametrizations of a neural network do not admit infinite-width limits that can learn features, which is crucial for pretraining and transfer learning such as with BERT. We propose simple modifications to the… CONTINUE READING

View PDF on arXiv

Save to Library

Create Alert

Cite

Launch Research Feed