Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

@inproceedings{Salimans2016WeightNA,
  title={Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks},
  author={Tim Salimans and Diederik P. Kingma},
  booktitle={NIPS},
  year={2016}
}
We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also… CONTINUE READING
Highly Influential
This paper has highly influenced a number of papers. REVIEW HIGHLY INFLUENTIAL CITATIONS
Highly Cited
This paper has 327 citations. REVIEW CITATIONS

7 Figures & Tables

Topics

Statistics

01002002015201620172018
Citations per Year

327 Citations

Semantic Scholar estimates that this publication has 327 citations based on the available data.

See our FAQ for additional information.