Scalable Kernel Methods via Doubly Stochastic Gradients


The general perception is that kernel methods are not scalable, so neural nets become the choice for large-scale nonlinear learning problems. Have we tried hard enough for kernel methods? In this paper, we propose an approach that scales up kernel methods using a novel concept called “doubly stochastic functional gradients”. Based on the fact that many kernel methods can be expressed as convex optimization problems, our approach solves the optimization problems by making two unbiased stochastic approximations to the functional gradient—one using random training points and another using random features associated with the kernel—and performing descent steps with this noisy functional gradient. Our algorithm is simple, need no commit to a preset number of random features, and allows the flexibility of the function class to grow as we see more incoming data in the streaming setting. We demonstrate that a function learned by this procedure after t iterations converges to the optimal function in the reproducing kernel Hilbert space in rate O(1/t), and achieves a generalization bound of O(1/ √ t). Our approach can readily scale kernel methods up to the regimes which are dominated by neural nets. We show competitive performances of our approach as compared to neural nets in datasets such as 2.3 million energy materials from MolecularSpace, 8 million handwritten digits from MNIST, and 1 million photos from ImageNet using convolution features.

View Slides

Extracted Key Phrases

1 Figure or Table

Citations per Year

96 Citations

Semantic Scholar estimates that this publication has 96 citations based on the available data.

See our FAQ for additional information.

Cite this paper

@inproceedings{Dai2014ScalableKM, title={Scalable Kernel Methods via Doubly Stochastic Gradients}, author={Bo Dai and Bo Xie and Niao He and Yingyu Liang and Anant Raj and Maria-Florina Balcan and Le Song}, booktitle={NIPS}, year={2014} }