Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity

@article{Schfer2021CompressionIA,
  title={Compression, inversion, and approximate PCA of dense kernel matrices at near-linear computational complexity},
  author={Florian Sch{\"a}fer and Timothy John Sullivan and Houman Owhadi},
  journal={Multiscale Model. Simul.},
  year={2021},
  volume={19},
  pages={688-730}
}
Dense kernel matrices Θ∈R^(N×N) obtained from point evaluations of a covariance function G at locations {x_i}1≤i≤N arise in statistics, machine learning, and numerical analysis. For covariance functions that are Green's functions elliptic boundary value problems and approximately equally spaced sampling points, we show how to identify a subset S⊂{1,…,N}×{1,…,N}, with #S=O(Nlog(N)log^d(N/ϵ)), such that the zero fill-in block-incomplete Cholesky decomposition of Θ_(i,j)1_((i,j)∈S) is an… 
View PDF on arXiv
45 Citations
Highly Influential Citations
1
Background Citations
20
Methods Citations
16
Results Citations
2

45 Citations

Kernel Flows: from learning kernels from data into the abyss
Sparse Cholesky factorization by Kullback-Leibler minimization
TLDR
An elimination ordering and sparsity pattern is proposed that allows us to compute ϵ-approximate inverse Cholesky factors of such Θ in computational complexity O(Nlog(N/ϵ)^d) in space and O( Nlog( N/δ)^(2d)) in time, which is the best asymptotic complexity for this class of problems.
Posterior and Computational Uncertainty in Gaussian Processes
TLDR
A new class of methods is developed that provides consistent estimation of the combined uncertainty arising from both the finite number of data observed and the finite amount of computation expended, and the consequences of ignoring computational uncertainty are demonstrated.
Samplets: A new paradigm for data compression
TLDR
The concept of samplets is introduced by transferring the construction of Tausch-White wavelets to the realm of data to obtain a multilevel representation of discrete data which directly enables data compression, detection of singularities and adaptivity.
Learning Partial Differential Equations in Reproducing Kernel Hilbert Spaces
TLDR
A new data-driven approach for learning the fundamental solutions of various linear partial differential equations (PDEs) given sample pairs of input-output functions by estimating the bestfit Green’s function and bias term in a reproducing kernel Hilbert space (RKHS) which allows to regularize their smoothness and impose various structural constraints.
Certified and fast computations with shallow covariance kernels
TLDR
This work introduces and analyze a new and certified algorithm for the low-rank approximation of a parameterized family of covariance operators which represents an extension of the adaptive cross approximation method for symmetric positive definite matrices.
Graph-Induced Rank Structures and their Representations
TLDR
The notion of a graph-induced rank structure (GIRS) which aims to capture the fine low rank structures which appear in sparse matrices and their inverses in terms of the adjacency graph $\mathbb{G}$.
De-noising by thresholding operator adapted wavelets Gene
Donoho and Johnstone [13] proposed a method from reconstructing an unknown smooth function u from noisy data u+ ζ by translating the empirical wavelet coefficients of u+ ζ towards zero. We consider…
Fast Eigenpairs Computation with Operator Adapted Wavelets and Hierarchical Subspace Correction
TLDR
The proposed algorithm is robust for the presence of multiple (a continuum of) scales and is shown to be of near-linear complexity when $\mathcal{L}$ is an (arbitrary local, e.g.~differential) operator mapping $\ mathscal{H}^s_0(\Omega)$ to $\mathCal{H}\Omega$ (e. g.~an elliptic PDE with rough coefficients).
Kernel Mode Decomposition and programmable/interpretable regression networks
TLDR
The proposed framework for programmable and interpretable regression networks for pattern recognition and address mode decomposition as a prototypical problem is introduced and the structure of some of these networks share intriguing similarities with convolutional neural networks while being interpretable, programable and amenable to theoretical analysis.
...
...

References

SHOWING 1-10 OF 174 REFERENCES
Compressing Rank-Structured Matrices via Randomized Sampling
TLDR
The proposed scheme is also useful in simplifying the implementation of certain operations on rank-structured matrices such as the matrix-matrix multiplication, low-rank update, addition, etc.
Data-sparse Approximation by Adaptive ℋ2-Matrices
TLDR
The basic ideas of ℋ- andℋ2-matrices are introduced and an algorithm that adaptively computes approximations of general matrices in the latter format is presented.
Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
TLDR
This work surveys and extends recent research which demonstrates that randomization offers a powerful tool for performing low-rank matrix approximation, and presents a modular framework for constructing randomized algorithms that compute partial matrix decompositions.
Approximation of eigenfunctions in kernel-based spaces
TLDR
This paper examines the corresponding eigenspaces of continuous kernels on compact domains and proves that they have optimality properties among all other subspaces of ℋ$\mathcal {H}$ and gives error bounds for the numerical approximation of the eIGensystem via suchSubspaces.
LU Factorization of Non-standard Forms and Direct Multiresolution Solvers
TLDR
This paper introduces the multiresolution LU factorization of non-standard forms (NS-forms) and develops fast directMultiresolution methods for solving systems of linear algebraic equations arising in elliptic problems and demonstrates how to construct the inverse in O ( N ) operations.
Localization of Matrix Factorizations
TLDR
A comprehensive framework is derived to rigorously answer the question of when and under what conditions the matrix factors inherit the localization of the original matrix for such fundamental matrix factorizations as the LU, QR, Cholesky, and polar factorizations.
Efficient SVM Training Using Low-Rank Kernel Representations
TLDR
This work shows that for a low rank kernel matrix it is possible to design a better interior point method (IPM) in terms of storage requirements as well as computational complexity and derives an upper bound on the change in the objective function value based on the approximation error and the number of active constraints (support vectors).
A Sparse Matrix Arithmetic Based on H-Matrices. Part I: Introduction to H-Matrices
TLDR
This paper is the first of a series and is devoted to the first introduction of the H-matrix concept, which allows the exact inversion of tridiagonal matrices.
Localization in Matrix Computations: Theory and Applications
Many important problems in mathematics and physics lead to (non-sparse) functions, vectors, or matrices in which the fraction of nonnegligible entries is vanishingly small compared the total number…
Randomized matrix-free trace and log-determinant estimators
TLDR
Random algorithms for estimating the trace and determinant of Hermitian positive semi-definite matrices and the error due to randomization are presented, for starting guesses whose elements are Gaussian or Rademacher random variables.
...
...