• Corpus ID: 227275332

Fast geometric learning with symbolic matrices

@inproceedings{Feydy2020FastGL,
  title={Fast geometric learning with symbolic matrices},
  author={Jean Feydy and Joan Alexis Glaun{\`e}s and Benjamin Charlier and Michael M. Bronstein},
  booktitle={Neural Information Processing Systems},
  year={2020}
}
Geometric methods rely on tensors that can be encoded using a symbolic formula and data arrays, such as kernel and distance matrices. We present an extension for standard machine learning frameworks that provides comprehensive support for this abstraction on CPUs and GPUs: our toolbox combines a versatile, transparent user interface with fast runtimes and low memory usage. Unlike general purpose acceleration frameworks such as XLA, our library turns generic Python code into binaries whose… 

Figures and Tables from this paper

Memory Safe Computations with XLA Compiler

An XLA compiler extension 1 is developed that adjusts the computational data-flow representation of an algorithm according to a user-specified memory limit and shows that k-nearest neighbour and sparse Gaussian process regression methods can be run at a much larger scale on a single device, where standard implementations would have failed.

Low-Precision Arithmetic for Fast Gaussian Processes

This approach improves the numerical stability and practical performance of conjugate gradients in low-precision over a wide range of settings, enabling GPs to train on 1.8 million data points in 10 hours on a single GPU, without requiring any sparse approximations.

Fast end-to-end learning on protein surfaces

A new framework for deep learning on protein structures that addresses limitations including the need to pre-compute the input features and mesh connectivities and achieves state-of-the-art performance with much faster run times and fewer parameters than previous models.

GNPM: Geometric-Aware Neural Parametric Models

We propose Geometric Neural Parametric Models (GNPM), a learned parametric model that takes into account the local structure of data to learn disentangled shape and pose latent spaces of 4D dynamics,

Unbalanced Optimal Transport, from Theory to Numerics

How unbalanced OT, entropic regularization and GW can work hand-in-hand to turn OT into efficient geometric loss functions for data sciences is explained.

Convex transport potential selection with semi-dual criterion

This work proposes to leverage the Brenier formulation of OT to perform Optimal Transport in a stochastic setting and can identify the potential that is closest to the true OT map between the source and the target and observes that this selected potential is not the one that performs best for the downstream transfer classification task.

Accurate Point Cloud Registration with Robust Optimal Transport

It is shown that recent OT solvers improve both optimization-based and deep learning methods for point cloud registration, boosting accuracy at an affordable computational cost and providing a new key method for the computer vision toolbox.

Neural Monge Map estimation and its applications

This work presents a scalable algorithm based on a weak form of the optimal transport problem, thus it only requires samples from the marginals instead of their analytic expres-sions, and can accommodate optimal transport between two distributions with different dimensions.

Sinkhorn Divergences for Unbalanced Optimal Transport

The formulation of Sinkhorn divergences is extended to the unbalanced setting of arbitrary positive measures, providing both theoretical and algorithmic advances, and a linear rate of convergence is shown, under mild assumptions, independent of the number of samples.

Bayesian Optimization with High-Dimensional Outputs

This work devise an efficient technique for exact multitask GP sampling that combines exploiting Kronecker structure in the covariance matrices with Matheron’s identity, allowing it to perform Bayesian Optimization using exact multi-task GP models with tens of thousands of correlated outputs.

References

SHOWING 1-10 OF 148 REFERENCES

Kernel Operations on the GPU, with Autodiff, without Memory Overflows

The KeOps library provides a fast and memory-efficient GPU support for tensors whose entries are given by a mathematical formula, such as kernel and distance matrices, including PyTorch CUDA tensors or the Halide and TVM libraries.

Fast Graph Representation Learning with PyTorch Geometric

PyTorch Geometric is introduced, a library for deep learning on irregularly structured input data such as graphs, point clouds and manifolds, built upon PyTorch, and a comprehensive comparative study of the implemented methods in homogeneous evaluation scenarios is performed.

Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs

Deep Graph Library (DGL) enables arbitrary message handling and mutation operators, flexible propagation rules, and is framework agnostic so as to leverage high-performance tensor, autograd operations, and other feature extraction modules already available in existing frameworks.

Mesh-TensorFlow: Deep Learning for Supercomputers

Mesh-TensorFlow is introduced, a language for specifying a general class of distributed tensor computations and used to implement an efficient data-parallel, model-Parallel version of the Transformer sequence-to-sequence model, surpassing state of the art results on WMT'14 English- to-French translation task and the one-billion-word language modeling benchmark.

TensorFlow: A system for large-scale machine learning

The TensorFlow dataflow model is described and the compelling performance that Tensor Flow achieves for several real-world applications is demonstrated.

GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration

This work presents an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM), a modified batched version of the conjugate gradients algorithm to derive all terms for training and inference in a single call.

Poincaré Embeddings for Learning Hierarchical Representations

This work introduces a new approach for learning hierarchical representations of symbolic data by embedding them into hyperbolic space -- or more precisely into an n-dimensional Poincare ball -- and introduces an efficient algorithm to learn the embeddings based on Riemannian optimization.

Geometric Deep Learning: Going beyond Euclidean data

Deep neural networks are used for solving a broad range of problems from computer vision, natural-language processing, and audio analysis where the invariances of these structures are built into networks used to model them.

Differentiable programming for image processing and deep learning in halide

This work extends the image processing language Halide with general reverse-mode automatic differentiation (AD), and the ability to automatically optimize the implementation of gradient computations, and shows how differentiable programming enables dramatically improving the quality of even traditional, feed-forward image processing algorithms, blurring the distinction between classical and deep methods.

Theano: A Python framework for fast computation of mathematical expressions

The performance of Theano is compared against Torch7 and TensorFlow on several machine learning models and recently-introduced functionalities and improvements are discussed.
...