On The Relative Error of Random Fourier Features for Preserving Kernel Distance

@article{Cheng2022OnTR,
  title={On The Relative Error of Random Fourier Features for Preserving Kernel Distance},
  author={Kuan Cheng and Shaofeng H.-C. Jiang and Luojian Wei and Zhide Wei},
  journal={ArXiv},
  year={2022},
  volume={abs/2210.00244}
}
The method of random Fourier features (RFF), proposed in a seminal paper by Rahimi and Recht (NIPS’07), is a powerful technique to find approximate low-dimensional representations of points in (high-dimensional) kernel space, for shift-invariant kernels. While RFF has been analyzed under various notions of error guarantee, the ability to preserve the kernel distance with relative error is less under-stood. We show that for a significant range of kernels, including the well-known Laplacian kernels… 
[PDF] Semantic Reader

Figures from this paper

References

SHOWING 1-10 OF 54 REFERENCES

Relative Error Embeddings of the Gaussian Kernel Distance

It is shown in this paper that for the Gaussian kernel the Euclidean norm between these mapped to features has $(1+\epsilon)-relative error with respect to the kernel distance.

Optimal Rates for Random Fourier Features

A detailed finite-sample theoretical analysis about the approximation quality of RFFs is provided by establishing optimal (in terms of the RFF dimension, and growing set size) performance guarantees in uniform norm, and presenting guarantees in Lr (1 ≤ r < ∞) norms.

On the Error of Random Fourier Features

The uniform error bound of that paper on random Fourier features is improved, as well as giving novel understandings of the embedding's variance, approximation error, and use in some machine learning methods.

Towards a Unified Analysis of Random Fourier Features

This work provides the first unified risk analysis of learning with random Fourier features using the squared error and Lipschitz continuous loss functions and devise a simple approximation scheme which provably reduces the computational cost without loss of statistical efficiency.

Near Input Sparsity Time Kernel Embeddings via Adaptive Sampling

A near input sparsity time algorithm for sampling the high-dimensional feature space implicitly defined by a kernel transformation, and shows how its subspace embedding bounds imply new statistical guarantees for kernel ridge regression.

The Geometry of Random Features

It is proved that for the case of many dimensions, the superiority of the orthogonal transform can be accurately measured by a property called the charm of the kernel, and that Orthogonal random features provide optimal (in terms of mean squared error) kernel estimators.

Low-Precision Random Fourier Features for Memory-Constrained Kernel Approximation

This work proposes using a low-precision quantization of random Fourier features (LP-RFFs) to build a high-rank approximation under a memory budget, and shows quantization has a negligible effect on generalization performance in important settings.

Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond

This survey systematically review the work on random features from the past ten years and discusses the relationship between random features and modern over-parameterized deep neural networks (DNNs), including the use of high dimensional random features in the analysis of DNNs as well as the gaps between current theoretical and empirical results.

Quasi-Monte Carlo Feature Maps for Shift-Invariant Kernels

A new discrepancy measure called box discrepancy is derived based on theoretical characterizations of the integration error with respect to a given sequence based on explicit box discrepancy minimization in Quasi-Monte Carlo (QMC) approximations.

Subspace Embeddings for the Polynomial Kernel

This work proposes the first fast oblivious subspace embeddings that are able to embed a space induced by a non-linear kernel without explicitly mapping the data to the high-dimensional space.
...