• Corpus ID: 231934118

Faster Kernel Matrix Algebra via Density Estimation

@article{Backurs2021FasterKM,
  title={Faster Kernel Matrix Algebra via Density Estimation},
  author={Arturs Backurs and Piotr Indyk and Cameron Musco and Tal Wagner},
  journal={ArXiv},
  year={2021},
  volume={abs/2102.08341}
}
We study fast algorithms for computing fundamental properties of a positive semidefinite kernel matrix K ∈ Rn×n corresponding to n points x1, . . . , xn ∈ R. In particular, we consider estimating the sum of kernel matrix entries, along with its top eigenvalue and eigenvector. We show that the sum of matrix entries can be estimated to 1 + relative error in time sublinear in n and linear in d for many popular kernels, including the Gaussian, exponential, and rational quadratic. For these kernels… 

Figures and Tables from this paper

Giga-scale Kernel Matrix Vector Multiplication on GPU
TLDR
A novel approximation procedure coined Faster-Fast and Free Memory Method (F 3 M) to address scaling issues of KMVM for tall and skinny data and demonstrates competitive results on Gaussian Process regression coupled with significant speedups on a variety of real-world datasets.
Sublinear Time Eigenvalue Approximation via Random Sampling
TLDR
The theoretical results are the first ones that can take advantage of the sparsity of A and require several new eigenvalue concentration and perturbation bounds for matrices with bounded entries, and numerical simulations demonstrate the effectiveness of the algorithms in practice.

References

SHOWING 1-10 OF 34 REFERENCES
ASKIT: Approximate Skeletonization Kernel-Independent Treecode in High Dimensions
TLDR
A fast algorithm for kernel summation problems in high-dimensions, which requires only kernel evaluations and does not use analytic expansions, and novel methods for pruning and approximating the far field are introduced.
Efficient Density Evaluation for Smooth Kernels
TLDR
This paper presents a collection of algorithms for efficient KDF evaluation under the assumptions that the kernel k is "smooth", i.e. the value changes at most polynomially with the distance, and presents a general reduction from density estimation to approximate near neighbor in the underlying space.
Improved Coresets for Kernel Density Estimates
TLDR
This work provides a careful analysis of the iterative Frank-Wolfe algorithm adapted to this context, an algorithm called kernel herding, which unites a broad line of work that spans statistics, machine learning, and geometry.
Recursive Sampling for the Nystrom Method
We give the first algorithm for kernel Nystrom approximation that runs in linear time in the number of training points and is provably accurate for all kernel matrices, without dependence on
Efficient SVM Training Using Low-Rank Kernel Representations
TLDR
This work shows that for a low rank kernel matrix it is possible to design a better interior point method (IPM) in terms of storage requirements as well as computational complexity and derives an upper bound on the change in the objective function value based on the approximation error and the number of active constraints (support vectors).
Sublinear Time Low-Rank Approximation of Distance Matrices
TLDR
A recursive algorithm based on additive projection-cost preserving sampling is developed and it is shown that for any underlying distance metric d, it is possible to achieve an additive error low rank approximation in sublinear time.
Improved fast gauss transform and efficient kernel density estimation
TLDR
An improved fast Gauss transform is developed to efficiently estimate sums of Gaussians in higher dimensions, where a new multivariate expansion scheme and an adaptive space subdivision technique dramatically improve the performance.
Dual-Tree Fast Gauss Transforms
TLDR
The extent to which the dual-tree recursion with finite-difference approximation can be integrated with multipole-like Hermite expansions in order to achieve reasonable efficiency across all bandwidth scales is explored, though only for low dimensionalities.
Kernel Density Estimation through Density Constrained Near Neighbor Search
TLDR
This paper gives a single data structure based on classical near neighbor search techniques that improves upon or essentially matches the query time and space complexity for all radial kernels considered in the literature so far, and shows how to improve both the query complexity and runtime by using recent advances in data-dependentNear neighbor search.
Hashing-Based-Estimators for Kernel Density in High Dimensions
TLDR
This work introduces a class of unbiased estimators for kernel density implemented through locality-sensitive hashing, and gives general theorems bounding the variance of such estimators.
...
...