Corpus ID: 5683093

Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks

@article{Huang2018OrthogonalWN,
  title={Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks},
  author={Lei Huang and Xianglong Liu and B. Lang and Adams Wei Yu and Bo Li},
  journal={ArXiv},
  year={2018},
  volume={abs/1709.06079}
}
Orthogonal matrix has shown advantages in training Recurrent Neural Networks (RNNs), but such matrix is limited to be square for the hidden-to-hidden transformation in RNNs. [...] Key Method We also propose a novel orthogonal weight normalization method to solve OMDSM. Particularly, it constructs orthogonal transformation over proxy parameters to ensure the weight matrix is orthogonal and back-propagates gradient information through the transformation during training.Expand
Controllable Orthogonalization in Training DNNs
  • Lei Huang, Li Liu, +4 authors L. Shao
  • Computer Science
  • 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2020
Projection Based Weight Normalization for Deep Neural Networks
CWY Parametrization for Scalable Learning of Orthogonal and Stiefel Matrices
Norm Loss: An efficient yet effective regularization method for deep neural networks
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 49 REFERENCES
Unitary Evolution Recurrent Neural Networks
Generalized BackPropagation, Étude De Cas: Orthogonality
Regularizing CNNs with Locally Constrained Decorrelations
...
1
2
3
4
5
...