• Corpus ID: 244709341

Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

@inproceedings{Birdal2021IntrinsicDP,
  title={Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks},
  author={Tolga Birdal and Aaron Lou and Leonidas J. Guibas and Umut cSimcsekli},
  booktitle={NeurIPS},
  year={2021}
}
This document supplements our main paper entitled Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks as follows: (i) Sec. S1 firsts gives some of the formal definitions and interpretations omitted from the main paper due to space limitations. Next, it involves a discussion and contrasts our dimension estimator against the commonly used ones. Finally, it provides additional details into the regularizer we devised in the main paper; (ii) we then provide the complement… 
Hausdorff dimension, heavy tails, and generalization in neural networks
TLDR
This paper proves generalization bounds for SGD under the assumption that its trajectories can be well-approximated by a Feller process, and shows that the generalization error can be controlled by the Hausdorff dimension of the trajectories, which is intimately linked to the tail behavior of the driving process.
Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms
TLDR
It is proved that the generalization error of a stochastic optimization algorithm can be bounded based on the ‘complexity’ of the fractal structure that underlies its invariant measure.
Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
TLDR
This study links compressibility to two recently established properties of SGD, and proves that the networks are guaranteed to be ‘`p-compressible’, and the compression errors of different pruning techniques become arbitrarily small as the network size increases.
Faster Optimization on Sparse Graphs via Neural Reparametrization
TLDR
This paper proposes a novel optimization method based on neural reparametrization: parametrizing the solution to the optimization problem with a graph neural network (GNN), leading to significant speed-up on large sparse graphs.
RipsNet: a general architecture for fast and robust estimation of the persistent homology of point clouds
TLDR
It is proved that RipsNet is robust to input perturbations in terms of the 1-Wasserstein distance, a major improvement over the standard computation of PDs that only enjoys Hausdorff stability, yielding Rips net to substantially outperform exactly-computed PDs in noisy settings.
Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithms
TLDR
This study proves novel generalization bounds through the lens of rate-distortion theory, and explicitly relate the concepts of mutual information, compressibility, and fractal dimensions in a single mathematical framework.
Machine Learning and Deep Learning -- A review for Ecologists
TLDR
A comprehensive overview of ML and DL is provided, starting with their historical developments, their algorithm families, their differences from traditional statistical tools, and universal ML principles, highlighting current and emerging applications for ecological problems.
Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learning
In this paper, we use tools from rate-distortion theory to establish new upper bounds on the generalization error of statistical distributed learning algorithms. Specifically, there are K clients
Topology and geometry of data manifold in deep learning
TLDR
The geometry of adversarial attacks in the classification task and spoofing attacks on face recognition systems are considered, and a method for assessing the generalizing ability of neural networks based on topological descriptors is proposed.

References

SHOWING 1-10 OF 88 REFERENCES
Connectivity-Optimized Representation Learning via Persistent Homology
TLDR
This work controls the connectivity of an autoencoder's latent space via a novel type of loss, operating on information from persistent homology, which is differentiable and presents a theoretical analysis of the properties induced by the loss.
A Topological Regularizer for Classifiers via Persistent Homology
TLDR
This paper proposes to enforce the structural simplicity of the classification boundary by regularizing over its topological complexity by measuring the importance of topological features in a meaningful manner, and provides a direct control over spurious topological structures.
Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology
TLDR
This work proposes neural persistence, a complexity measure for neural network architectures based on topological data analysis on weighted stratified graphs and derives a neural persistence-based stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss.
Cycle Registration in Persistent Homology with Applications in Topological Bootstrap
TLDR
A novel approach for comparing the persistent homology representations of two spaces (filtrations) by defining a correspondence relation between individual persistent cycles of two different spaces, and devising a method for computing this correspondence.
A Topology Layer for Machine Learning
TLDR
A differentiable topology layer that computes persistent homology based on level set Filtrations and distance-bases filtrations is presented that can serve as a regularizer directly on data or the weights of machine learning models.
Hausdorff dimension, heavy tails, and generalization in neural networks
TLDR
This paper proves generalization bounds for SGD under the assumption that its trajectories can be well-approximated by a Feller process, and shows that the generalization error can be controlled by the Hausdorff dimension of the trajectories, which is intimately linked to the tail behavior of the driving process.
Measuring the Intrinsic Dimension of Objective Landscapes
TLDR
Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where it is concluded that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10.
Intrinsic dimension of data representations in deep neural networks
TLDR
The intrinsic dimensionality of data-representations is studied, i.e. the minimal number of parameters needed to describe a representation, and it is found that, in a trained network, the ID is orders of magnitude smaller than the number of units in each layer.
Understanding deep learning (still) requires rethinking generalization
TLDR
These experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data, and corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity.
Generalization bounds via distillation
TLDR
This paper theoretically investigates the following empirical phenomenon: given a high-complexity network with poor generalization bounds, one can distill it into a network with nearly identical predictions but low complexity and vastly smaller generalization limits, as well as a variety of experiments demonstrating similar generalization performance between the original network and its distillation.
...
...