• Corpus ID: 52913094

McTorch, a manifold optimization library for deep learning

@article{Meghwanshi2018McTorchAM,
  title={McTorch, a manifold optimization library for deep learning},
  author={Mayank Meghwanshi and Pratik Jawanpuria and Anoop Kunchukuttan and Hiroyuki Kasai and Bamdev Mishra},
  journal={ArXiv},
  year={2018},
  volume={abs/1810.01811}
}
In this paper, we introduce McTorch, a manifold optimization library for deep learning that extends PyTorch. It aims to lower the barrier for users wishing to use manifold constraints in deep learning applications, i.e., when the parameters are constrained to lie on a manifold. Such constraints include the popular orthogonality and rank constraints, and have been recently used in a number of applications in deep learning. McTorch follows PyTorch's architecture and decouples manifold definitions… 
[PDF] Semantic Reader
34 Citations
Highly Influential Citations
2
Background Citations
13
Methods Citations
10

34 Citations

NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learning

A new software package called NCVX is introduced, whose initial release contains the solver PyGRANSO, a PyTorch-enabled general-purpose optimization package for constrained machine/deep learning problems, the first of its kind.

Manifold optimization for optimal transport

This work discusses optimization-related ingredients that allow modeling the OT problem on smooth Riemannian manifolds by exploiting the geometry of the search space and makes available the Manifold optimization-based Optimal Transport repository with codes useful in solving OT problems in Python and Matlab.

A Dual Framework for Low-rank Tensor Completion

This work proposes a variant of the latent trace norm that helps in learning a non-sparse combination of tensors, and develops a dual framework for solving the low-rank tensor completion problem.

Manifold optimization for non-linear optimal transport problems

This work discusses optimization-related ingredients that allow modeling the OT problem on smooth Riemannian manifolds by exploiting the geometry of the search space and makes available the Manifold optimization-based Optimal Transport repository, or MOT, repository with codes useful in solving OT problems in Python and Matlab.

Sliced Gromov-Wasserstein

A novel OT discrepancy is defined that can deal with large scale distributions via a slicing approach and is demonstrated to have ability to tackle similar problems as GW while being several order of magnitudes faster to compute.

TAOTF: A Two-stage Approximately Orthogonal Training Framework in Deep Neural Networks

A novel two-stage approximately orthogonal training framework (TAOTF) to solve the problem of robustness to noisy data of Deep Neural Network models with constraints to solve this problem in noisy data scenarios and achieves stable and superior performances to existing methods.

Riemannian Hamiltonian methods for min-max optimization on manifolds

The proposed Riemannian Hamiltonian methods (RHM) are extended to include consensus regularization and to the stochastic setting and illustrate the efficacy of the proposed RHM in applications such as subspace robust Wasserstein distance, robust training of neural networks, and generative adversarial networks.

Geometry-aware domain adaptation for unsupervised alignment of word embeddings

A novel manifold based geometric approach for learning unsupervised alignment of word embeddings between the source and the target languages by formulating the alignment learning problem as a domain adaptation problem over the manifold of doubly stochastic matrices.

Dissolving Constraints for Riemannian Optimization

The theoretical properties of CDF are studied and it is proved that the original problem and CDF have the same first-order and second-order stationary points, local minimizers, and Łojasiewicz exponents in a neighborhood of the feasible region.

TpG Geoopt: Riemannian Optimization in PyTorch

The core of Geoopt is a standard Manifold interface that allows for the generic implementation of optimization algorithms and several algorithms and arithmetic methods for supported manifolds, which allow composing geometry-aware neural network layers that can be integrated with existing models.

References

SHOWING 1-10 OF 34 REFERENCES

Geometry Aware Constrained Optimization Techniques for Deep Learning

This paper generalizes the Stochastic Gradient Descent (SGD) and RMSProp algorithms to the setting of Riemannian optimization, and substantiates their proposed extensions with a range of relevant problems in machine learning such as incremental Principal Component Analysis, computating the RiemANNian centroids of SPD matrices, and Deep Metric Learning.

A Unified Framework for Structured Low-rank Matrix Learning

We propose a novel optimization framework for learning a low-rank matrix which is also constrained to lie in a linear subspace. Exploiting the duality theory, we present a factorization that…

Building Deep Networks on Grassmann Manifolds

This paper proposes a deep network architecture by generalizing the Euclidean network paradigm to Grassmann manifolds and designs full rank mapping layers to transform input Grassmannian data to more desirable ones, and exploits re-orthonormalization layers to normalize the resulting matrices.

A Dual Framework for Low-rank Tensor Completion

This work proposes a variant of the latent trace norm that helps in learning a non-sparse combination of tensors, and develops a dual framework for solving the low-rank tensor completion problem.

Automatic differentiation in PyTorch

An automatic differentiation module of PyTorch is described — a library designed to enable rapid research on machine learning models that focuses on differentiation of purely imperative programs, with a focus on extensibility and low overhead.

Symmetry-invariant optimization in deep networks

This work shows that commonly used deep networks, such as those which use a max-pooling and sub-sampling layer, possess more complex forms of symmetry arising from scaling based reparameterization of the network weights.

Manopt, a matlab toolbox for optimization on manifolds

The Manopt toolbox, available at www.manopt.org, is a user-friendly, documented piece of software dedicated to simplify experimenting with state of the art Riemannian optimization algorithms, which aims particularly at lowering the entrance barrier.

MADMM: A Generic Algorithm for Non-smooth Optimization on Manifolds

This paper proposes the Manifold Alternating Directions Method of Multipliers (MADMM), an extension of the classical ADMM scheme for manifold-constrained non-smooth optimization problems, and is the first generic non-Smooth manifold optimization method.

Low-rank tensor completion: a Riemannian manifold preconditioning approach

A novel Riemannian manifold preconditioning approach for the tensor completion problem with rank constraint that exploits the least-squares structure of the cost function and takes into account the structured symmetry that exists in Tucker decomposition.

Linear Regression under Fixed-Rank Constraints: A Riemannian Approach

Numerical experiments on benchmarks suggest that the proposed algorithms compete with the state-of-the-art, and that manifold optimization offers a versatile framework for the design of rank-constrained machine learning algorithms.