VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning

@article{Shang2020VRSGDAS,
  title={VR-SGD: A Simple Stochastic Variance Reduction Method for Machine Learning},
  author={F. Shang and Kaiwen Zhou and Hongying Liu and James Cheng and I. Tsang and L. Zhang and D. Tao and Licheng Jiao},
  journal={IEEE Transactions on Knowledge and Data Engineering},
  year={2020},
  volume={32},
  pages={188-202}
}
  • F. Shang, Kaiwen Zhou, +5 authors Licheng Jiao
  • Published 2020
  • Computer Science, Mathematics
  • IEEE Transactions on Knowledge and Data Engineering
  • In this paper, we propose a simple variant of the original SVRG, called variance reduced stochastic gradient descent (VR-SGD. [...] Key Method We also design two different update rules for smooth and non-smooth objective functions, respectively, which means that VR-SGD can tackle non-smooth and/or non-strongly convex problems directly without any reduction techniques. Moreover, we analyze the convergence properties of VR-SGD for strongly convex problems, which show that VR-SGD attains linear convergence…Expand Abstract
    ASVRG: Accelerated Proximal SVRG
    • 13
    • PDF
    Guaranteed Sufficient Decrease for Stochastic Variance Reduced Gradient Optimization
    • 6
    • PDF
    L-SVRG and L-Katyusha with Arbitrary Sampling
    • 11
    • PDF
    A Stochastic Variance Reduced Extragradient Method for Sparse Machine Learning Problems
    Accelerating SGD using flexible variance reduction on large-scale datasets
    • 2
    SAAGs: Biased stochastic variance reduction methods for large-scale learning
    • 3
    • Highly Influenced
    • PDF
    SAAGs: Biased Stochastic Variance Reduction Methods
    • 2
    • Highly Influenced
    Efficient Relaxed Gradient Support Pursuit for Sparsity Constrained Non-convex Optimization
    Accelerated Incremental Gradient Descent using Momentum Acceleration with Scaling Factor
    • 2
    • PDF

    References

    Publications referenced by this paper.
    SHOWING 1-10 OF 95 REFERENCES
    On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants
    • 153
    • PDF
    Barzilai-Borwein Step Size for Stochastic Gradient Descent
    • 80
    • PDF
    Fast Stochastic Variance Reduced Gradient Method with Momentum Acceleration for Machine Learning
    • 19
    • PDF
    Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
    • 183
    • Highly Influential
    • PDF
    Variance-Reduced and Projection-Free Stochastic Optimization
    • 103
    • PDF
    Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
    • 425
    • Highly Influential
    • PDF
    Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
    • 1,534
    • PDF
    Accelerated Gradient Methods for Stochastic Optimization and Online Learning
    • 153
    • Highly Influential
    • PDF
    Guaranteed Sufficient Decrease for Stochastic Variance Reduced Gradient Optimization
    • 6
    • PDF
    Adam: A Method for Stochastic Optimization
    • 49,762
    • PDF