• Corpus ID: 238583457

Optimal Gradient Tracking for Decentralized Optimization

  title={Optimal Gradient Tracking for Decentralized Optimization},
  author={Zhuoqing Song and Lei Shi and Shi Pu and Ming Yan},
In this paper, we focus on solving the decentralized optimization problem of minimizing the sum of n objective functions over a multi-agent network. The agents are embedded in an undirected graph where they can only send/receive information directly to/from their immediate neighbors. Assuming smooth and strongly convex objective functions, we propose an Optimal Gradient Tracking (OGT) method that achieves the optimal gradient computation complexity O (√ κ log 1 ǫ ) and the optimal communication… 

Figures and Tables from this paper

Distributed Saddle-Point Problems: Lower Bounds, Near-Optimal and Robust Algorithms
This paper presents a new federated algorithm for cenralized distributed saddle point problems – Extra Step Local SGD and shows the effectiveness of this method in practice, which trains GANs in a distributed manner.
Recent theoretical advances in decentralized distributed convex optimization.
This paper focuses on how the results of decentralized distributed convex optimization can be explained based on optimal algorithms for the non-distributed setup, and provides recent results that have not been published yet.


Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks
The efficiency of MSDA against state-of-the-art methods for two problems: least-squares regression and classification by logistic regression is verified.
Push–Pull Gradient Methods for Distributed Optimization in Networks
“push–pull” is the first class of algorithms for distributed optimization over directed graphs for strongly convex and smooth objective functions over a network and outperform other existing linearly convergent schemes, especially for ill-conditioned problems and networks that are not well balanced.
Achieving Linear Convergence in Distributed Asynchronous Multiagent Optimization
This article proposes a general distributed asynchronous algorithmic framework whereby agents can update their local variables as well as communicate with their neighbors at any time, without any form of coordination, and proves that this is the first distributed algorithm with provable geometric convergence rate in such a general asynchronous setting.
Multi-consensus Decentralized Accelerated Gradient Descent
A novel algorithm is proposed that can achieve near optimal communication complexity, matching the known lower bound up to a logarithmic factor of the condition number of the problem.
Fast Distributed Gradient Methods
This work proposes two fast distributed gradient algorithms based on the centralized Nesterov gradient algorithm and establishes their convergence rates in terms of the per-node communications K and theper-node gradient evaluations k.
Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
The error due to limits in communication resources decreases at a fast rate even in the case of non-strongly-convex objective functions, and the first optimal first-order decentralized algorithm called multi-step primal-dual (MSPD) and its corresponding optimal convergence rate are provided.
Convergence of Asynchronous Distributed Gradient Methods Over Stochastic Networks
A new asynchronous distributed gradient method (AsynDGM) based on consensus theory is developed that not only allows for asynchronous implementation in a completely distributed manner but also is able to seek the exact optimum even with constant stepsizes.
Harnessing smoothness to accelerate distributed optimization
  • Guannan Qu, Na Li
  • Computer Science
    2016 IEEE 55th Conference on Decision and Control (CDC)
  • 2016
This paper proposes a distributed algorithm that, despite using the same amount of communication per iteration as DGD, can effectively harnesses the function smoothness and converge to the optimum with a rate of O(1/t) if the objective function is strongly convex and smooth.
Augmented distributed gradient methods for multi-agent optimization under uncoordinated constant stepsizes
A new augmented distributed gradient method (termed Aug-DGM) based on consensus theory is developed that will be able to seek the exact optimum even with constant stepsizes assuming that the global objective function has Lipschitz gradient.
Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis
A by-product of this analysis is a tuning recommendation for several existing (non-accelerated) distributed algorithms yielding provably faster (worst-case) convergence rate for the class of problems under consideration.