• Corpus ID: 211677691

Differentiating through the Fr\'echet Mean

@article{Lou2020DifferentiatingTT,
  title={Differentiating through the Fr\'echet Mean},
  author={Aaron Lou and Isay Katsman and Qingxuan Jiang and Serge J. Belongie and Ser-Nam Lim and Christopher De Sa},
  journal={arXiv: Machine Learning},
  year={2020}
}
Recent advances in deep representation learning on Riemannian manifolds extend classical deep learning operations to better capture the geometry of the manifold. One possible extension is the Frechet mean, the generalization of the Euclidean mean; however, it has been difficult to apply because it lacks a closed form with an easily computable derivative. In this paper, we show how to differentiate through the Frechet mean for arbitrary Riemannian manifolds. Then, focusing on hyperbolic space… 

Figures and Tables from this paper

Hyperbolic Deep Neural Networks: A Survey
TLDR
A coherent and a comprehensive review of the literature around the neural components in the construction of HDNN, as well as the generalization of the leading deep approaches to the hyperbolic space are presented.
Learning Hyperbolic Representations for Unsupervised 3D Segmentation
TLDR
This work proposes learning effective representations of 3D patches for unsupervised segmentation through a variational autoencoder (VAE) with a hyperbolic latent space and a proposed gyroplane convolutional layer, which better models the underlying hierarchical structure within a 3D image.
Capturing implicit hierarchical structure in 3D biomedical images with self-supervised hyperbolic representations
TLDR
This work considers encoder-decoder architectures with a hyperbolic latent space, to explicitly capture hierarchical relationships present in subvolumes of the data, and introduces an essential self-supervised loss—in addition to the standard VAE loss—which infers approximate hierarchies.
Capturing implicit hierarchical structure in 3D biomedical images with self-supervised hyperbolic representations
TLDR
This work considers encoder-decoder architectures with a hyperbolic latent space, to explicitly capture hierarchical relationships present in subvolumes of the data, and introduces an essential self-supervised loss—in addition to the standard VAE loss—which infers approximate hierarchies.

References

SHOWING 1-10 OF 41 REFERENCES
Riemannian batch normalization for SPD neural networks
TLDR
A Riemannian batch normalization (batch- norm) algorithm is introduced, which generalizes the one used in Euclidean nets and derives a new manifold-constrained gradient descent algorithm working in the space of SPD matrices, allowing to learn the batchnorm layer.
Hyperbolic Neural Networks
TLDR
This work combines the formalism of Mobius gyrovector spaces with the Riemannian geometry of the Poincare model of hyperbolic spaces to derivehyperbolic versions of important deep learning tools: multinomial logistic regression, feed-forward and recurrent neural networks such as gated recurrent units.
Lorentzian Distance Learning for Hyperbolic Representations
TLDR
An approach to learn representations based on the Lorentzian distance in hyperbolic geometry, which makes it appropriate to represent hierarchies where parent nodes minimize the distances to their descendants and have smaller Euclidean norm than their children.
Trivializations for Gradient-Based Optimization on Manifolds
TLDR
This work proves a formula for the gradient of the exponential of matrices, which can be of practical interest on its own and shows how dynamic trivializations improve the performance of existing methods on standard tasks designed to test long-term memory within neural networks.
Hyperbolic Graph Convolutional Neural Networks
TLDR
This work proposes Hyperbolic Graph Convolutional Neural Network (HGCN), the first inductive hyperbolic GCN that leverages both the expressiveness of GCNs andhyperbolic geometry to learn inductive node representations for hierarchical and scale-free graphs.
Poincaré Embeddings for Learning Hierarchical Representations
TLDR
This work introduces a new approach for learning hierarchical representations of symbolic data by embedding them into hyperbolic space -- or more precisely into an n-dimensional Poincare ball -- and introduces an efficient algorithm to learn the embeddings based on Riemannian optimization.
Poincaré GloVe: Hyperbolic Word Embeddings
TLDR
Empirically, based on extensive experiments, it is proved that the embeddings, trained unsupervised, are the first to simultaneously outperform strong and popular baselines on the tasks of similarity, analogy and hypernymy detection.
Hyperbolic Attention Networks
TLDR
This work introduces hyperbolic attention networks to endow neural networks with enough capacity to match the complexity of data with hierarchical and power-law structure and re-expressing the ubiquitous mechanism of soft attention in terms of operations defined for hyperboloid and Klein models.
OptNet: Differentiable Optimization as a Layer in Neural Networks
TLDR
OptNet is presented, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks, and shows how techniques from sensitivity analysis, bilevel optimization, and implicit differentiation can be used to exactly differentiate through these layers.
Differentiable Convex Optimization Layers
TLDR
This paper introduces disciplined parametrized programming, a subset of disciplined convex programming, and demonstrates how to efficiently differentiate through each of these components, allowing for end-to-end analytical differentiation through the entire convex program.
...
...