SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization

Abstract

We propose a new algorithm for minimizing reg-ularized empirical loss: Stochastic Dual Newton Ascent (SDNA). Our method is dual in nature: in each iteration we update a random subset of the dual variables. However, unlike existing methods such as stochastic dual coordinate ascent, SDNA is capable of utilizing all local curvature information contained in the examples, which leads to striking improvements in both theory and practice – sometimes by orders of magnitude. In the special case when an L2-regularizer is used in the primal, the dual problem is a concave quadratic maximization problem plus a separable term. In this regime, SDNA in each step solves a prox-imal subproblem involving a random principal submatrix of the Hessian of the quadratic function ; whence the name of the method.

Extracted Key Phrases

5 Figures and Tables

Showing 1-10 of 12 references

Robust block coordinate descent

  • Kimon Fountoulakis, Rachael Tappenden
  • 2014

S2GD: Semistochastic gradient descent methods

  • Jakub Konečn´konečn´y, Peter Richtárik
  • 2014

Semistochastic coordinate descent

  • Konečn´konečn´y, Jakub, Qu, Zheng, Peter Richtárik
  • 2014
0102020162017
Citations per Year

Citation Acceleration: +13%

This publication has seen a change of +13% in citations over the last two years.

Learn more about how we calculate this metric in our FAQ.