• Corpus ID: 238583457

Optimal Gradient Tracking for Decentralized Optimization

@inproceedings{Song2021OptimalGT,
  title={Optimal Gradient Tracking for Decentralized Optimization},
  author={Zhuoqing Song and Lei Shi and Shi Pu and Ming Yan},
  year={2021}
}
In this paper, we focus on solving the decentralized optimization problem of minimizing the sum of n objective functions over a multi-agent network. The agents are embedded in an undirected graph where they can only send/receive information directly to/from their immediate neighbors. Assuming smooth and strongly convex objective functions, we propose an Optimal Gradient Tracking (OGT) method that achieves the optimal gradient computation complexity O (√ κ log 1 ǫ ) and the optimal communication… 

Figures and Tables from this paper

Recent theoretical advances in decentralized distributed convex optimization.
TLDR
This paper focuses on how the results of decentralized distributed convex optimization can be explained based on optimal algorithms for the non-distributed setup, and provides recent results that have not been published yet.
Distributed Saddle-Point Problems: Lower Bounds, Near-Optimal and Robust Algorithms
TLDR
This paper presents a new federated algorithm for cenralized distributed saddle point problems – Extra Step Local SGD and shows the effectiveness of this method in practice, which trains GANs in a distributed manner.
Decentralized Saddle-Point Problems with Different Constants of Strong Convexity and Strong Concavity
TLDR
This paper study distributed saddle-point problems (SPP) with strongly-convex-strongly-concave smooth objectives that have different strong convexity and strong concavity parameters of composite terms, which correspond to min and max variables, and bilinear saddle- point part.

References

SHOWING 1-10 OF 68 REFERENCES
Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks
TLDR
The efficiency of MSDA against state-of-the-art methods for two problems: least-squares regression and classification by logistic regression is verified.
Push–Pull Gradient Methods for Distributed Optimization in Networks
TLDR
“push–pull” is the first class of algorithms for distributed optimization over directed graphs for strongly convex and smooth objective functions over a network and outperform other existing linearly convergent schemes, especially for ill-conditioned problems and networks that are not well balanced.
Achieving Linear Convergence in Distributed Asynchronous Multiagent Optimization
TLDR
This article proposes a general distributed asynchronous algorithmic framework whereby agents can update their local variables as well as communicate with their neighbors at any time, without any form of coordination, and proves that this is the first distributed algorithm with provable geometric convergence rate in such a general asynchronous setting.
Multi-consensus Decentralized Accelerated Gradient Descent
TLDR
A novel algorithm is proposed that can achieve near optimal communication complexity, matching the known lower bound up to a logarithmic factor of the condition number of the problem.
Fast Distributed Gradient Methods
TLDR
This work proposes two fast distributed gradient algorithms based on the centralized Nesterov gradient algorithm and establishes their convergence rates in terms of the per-node communications K and theper-node gradient evaluations k.
Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
TLDR
The error due to limits in communication resources decreases at a fast rate even in the case of non-strongly-convex objective functions, and the first optimal first-order decentralized algorithm called multi-step primal-dual (MSPD) and its corresponding optimal convergence rate are provided.
An Accelerated Method For Decentralized Distributed Stochastic Optimization Over Time-Varying Graphs
TLDR
This work proposes the first accelerated (in the sense of Nesterov’s acceleration) method that simultaneously attains optimal up to a logarithmic factor communication and oracle complexity bounds for smooth strongly convex distributed stochastic optimization.
Convergence of Asynchronous Distributed Gradient Methods Over Stochastic Networks
TLDR
A new asynchronous distributed gradient method (AsynDGM) based on consensus theory is developed that not only allows for asynchronous implementation in a completely distributed manner but also is able to seek the exact optimum even with constant stepsizes.
Optimal and Practical Algorithms for Smooth and Strongly Convex Decentralized Optimization
TLDR
This work proposes two new algorithms for the task of decentralized minimization of the sum of smooth strongly convex functions stored across the nodes a network and presents them as accelerated variants of the Forward Backward algorithm to solve monotone inclusions associated with the decentralized optimization problem.
Harnessing smoothness to accelerate distributed optimization
  • Guannan Qu, Na Li
  • Computer Science
    2016 IEEE 55th Conference on Decision and Control (CDC)
  • 2016
TLDR
This paper proposes a distributed algorithm that, despite using the same amount of communication per iteration as DGD, can effectively harnesses the function smoothness and converge to the optimum with a rate of O(1/t) if the objective function is strongly convex and smooth.
...
...