McTorch, a manifold optimization library for deep learning
@article{Meghwanshi2018McTorchAM, title={McTorch, a manifold optimization library for deep learning}, author={Mayank Meghwanshi and Pratik Jawanpuria and Anoop Kunchukuttan and Hiroyuki Kasai and Bamdev Mishra}, journal={ArXiv}, year={2018}, volume={abs/1810.01811} }
In this paper, we introduce McTorch, a manifold optimization library for deep learning that extends PyTorch. It aims to lower the barrier for users wishing to use manifold constraints in deep learning applications, i.e., when the parameters are constrained to lie on a manifold. Such constraints include the popular orthogonality and rank constraints, and have been recently used in a number of applications in deep learning. McTorch follows PyTorch's architecture and decouples manifold definitions…
34 Citations
NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learning
- Computer ScienceArXiv
- 2022
A new software package called NCVX is introduced, whose initial release contains the solver PyGRANSO, a PyTorch-enabled general-purpose optimization package for constrained machine/deep learning problems, the first of its kind.
Manifold optimization for optimal transport
- Computer Science, MathematicsArXiv
- 2021
This work discusses optimization-related ingredients that allow modeling the OT problem on smooth Riemannian manifolds by exploiting the geometry of the search space and makes available the Manifold optimization-based Optimal Transport repository with codes useful in solving OT problems in Python and Matlab.
A Dual Framework for Low-rank Tensor Completion
- Computer ScienceNeurIPS
- 2018
This work proposes a variant of the latent trace norm that helps in learning a non-sparse combination of tensors, and develops a dual framework for solving the low-rank tensor completion problem.
Manifold optimization for non-linear optimal transport problems
- Computer Science, Mathematics
- 2021
This work discusses optimization-related ingredients that allow modeling the OT problem on smooth Riemannian manifolds by exploiting the geometry of the search space and makes available the Manifold optimization-based Optimal Transport repository, or MOT, repository with codes useful in solving OT problems in Python and Matlab.
Sliced Gromov-Wasserstein
- Computer ScienceNeurIPS
- 2019
A novel OT discrepancy is defined that can deal with large scale distributions via a slicing approach and is demonstrated to have ability to tackle similar problems as GW while being several order of magnitudes faster to compute.
TAOTF: A Two-stage Approximately Orthogonal Training Framework in Deep Neural Networks
- Computer ScienceArXiv
- 2022
A novel two-stage approximately orthogonal training framework (TAOTF) to solve the problem of robustness to noisy data of Deep Neural Network models with constraints to solve this problem in noisy data scenarios and achieves stable and superior performances to existing methods.
Riemannian Hamiltonian methods for min-max optimization on manifolds
- Computer ScienceArXiv
- 2022
The proposed Riemannian Hamiltonian methods (RHM) are extended to include consensus regularization and to the stochastic setting and illustrate the efficacy of the proposed RHM in applications such as subspace robust Wasserstein distance, robust training of neural networks, and generative adversarial networks.
Geometry-aware domain adaptation for unsupervised alignment of word embeddings
- Computer ScienceACL
- 2020
A novel manifold based geometric approach for learning unsupervised alignment of word embeddings between the source and the target languages by formulating the alignment learning problem as a domain adaptation problem over the manifold of doubly stochastic matrices.
Dissolving Constraints for Riemannian Optimization
- Computer Science
- 2022
The theoretical properties of CDF are studied and it is proved that the original problem and CDF have the same first-order and second-order stationary points, local minimizers, and Łojasiewicz exponents in a neighborhood of the feasible region.
TpG Geoopt: Riemannian Optimization in PyTorch
- Computer Science
- 2020
The core of Geoopt is a standard Manifold interface that allows for the generic implementation of optimization algorithms and several algorithms and arithmetic methods for supported manifolds, which allow composing geometry-aware neural network layers that can be integrated with existing models.
References
SHOWING 1-10 OF 34 REFERENCES
Geometry Aware Constrained Optimization Techniques for Deep Learning
- Computer Science2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- 2018
This paper generalizes the Stochastic Gradient Descent (SGD) and RMSProp algorithms to the setting of Riemannian optimization, and substantiates their proposed extensions with a range of relevant problems in machine learning such as incremental Principal Component Analysis, computating the RiemANNian centroids of SPD matrices, and Deep Metric Learning.
A Unified Framework for Structured Low-rank Matrix Learning
- Computer ScienceICML
- 2018
We propose a novel optimization framework for learning a low-rank matrix which is also constrained to lie in a linear subspace. Exploiting the duality theory, we present a factorization that…
Building Deep Networks on Grassmann Manifolds
- Computer ScienceAAAI
- 2018
This paper proposes a deep network architecture by generalizing the Euclidean network paradigm to Grassmann manifolds and designs full rank mapping layers to transform input Grassmannian data to more desirable ones, and exploits re-orthonormalization layers to normalize the resulting matrices.
A Dual Framework for Low-rank Tensor Completion
- Computer ScienceNeurIPS
- 2018
This work proposes a variant of the latent trace norm that helps in learning a non-sparse combination of tensors, and develops a dual framework for solving the low-rank tensor completion problem.
Automatic differentiation in PyTorch
- Computer Science
- 2017
An automatic differentiation module of PyTorch is described — a library designed to enable rapid research on machine learning models that focuses on differentiation of purely imperative programs, with a focus on extensibility and low overhead.
Symmetry-invariant optimization in deep networks
- Computer ScienceArXiv
- 2015
This work shows that commonly used deep networks, such as those which use a max-pooling and sub-sampling layer, possess more complex forms of symmetry arising from scaling based reparameterization of the network weights.
Manopt, a matlab toolbox for optimization on manifolds
- Computer ScienceJ. Mach. Learn. Res.
- 2014
The Manopt toolbox, available at www.manopt.org, is a user-friendly, documented piece of software dedicated to simplify experimenting with state of the art Riemannian optimization algorithms, which aims particularly at lowering the entrance barrier.
MADMM: A Generic Algorithm for Non-smooth Optimization on Manifolds
- Computer ScienceECCV
- 2016
This paper proposes the Manifold Alternating Directions Method of Multipliers (MADMM), an extension of the classical ADMM scheme for manifold-constrained non-smooth optimization problems, and is the first generic non-Smooth manifold optimization method.
Low-rank tensor completion: a Riemannian manifold preconditioning approach
- Computer ScienceICML
- 2016
A novel Riemannian manifold preconditioning approach for the tensor completion problem with rank constraint that exploits the least-squares structure of the cost function and takes into account the structured symmetry that exists in Tucker decomposition.
Linear Regression under Fixed-Rank Constraints: A Riemannian Approach
- Computer ScienceICML
- 2011
Numerical experiments on benchmarks suggest that the proposed algorithms compete with the state-of-the-art, and that manifold optimization offers a versatile framework for the design of rank-constrained machine learning algorithms.