Distributed Optimization, Averaging via ADMM, and Network Topology

@article{Frana2020DistributedOA,
  title={Distributed Optimization, Averaging via ADMM, and Network Topology},
  author={Guilherme França and Jos{\'e} Bento},
  journal={Proceedings of the IEEE},
  year={2020},
  volume={108},
  pages={1939-1952}
}
There has been an increasing necessity for scalable optimization methods, especially due to the explosion in the size of data sets and model complexity in modern machine learning applications. Scalable solvers often distribute the computation over a network of processing units. For simple algorithms, such as gradient descent, the dependence of the convergence time with the topology of this network is well known. However, for more involved algorithms, such as the alternating direction method of… 

Figures and Tables from this paper

References

SHOWING 1-10 OF 56 REFERENCES
How is Distributed ADMM Affected by Network Topology
TLDR
A full characterization of the convergence of distributed over-relaxed ADMM for the same type of consensus problem in terms of the topology of the underlying graph is provided and a proof of the aforementioned conjecture is shown it is valid for any graph, even the ones whose random walks cannot be accelerated via Markov chain lifting.
Convergence Rate of Distributed ADMM Over Networks
TLDR
A new distributed algorithm based on alternating direction method of multipliers (ADMM) to minimize sum of locally known convex functions using communication over a network and highlights the effect of network and communication weights on the convergence rate through degrees of the nodes, the smallest nonzero eigenvalue, and operator norm of the communication matrix.
Distributed Averaging Via Lifted Markov Chains
TLDR
This paper designs an algorithm with the fastest possible rate of convergence using a nonreversible Markov chain on the given network graph using the Metropolis-Hastings method, and provides the fastest mixingMarkov chain given the network topological constraints.
Polynomial Filtering for Fast Convergence in Distributed Consensus
TLDR
This paper proposes to accelerate the convergence rate for given network matrices by the use of polynomial filtering algorithms, and forms the computation of the coefficients of the optimal polynometric as a semidefinite program that can be efficiently and globally solved for both static and dynamic network topologies.
Optimal Algorithms for Smooth and Strongly Convex Distributed Optimization in Networks
TLDR
The efficiency of MSDA against state-of-the-art methods for two problems: least-squares regression and classification by logistic regression is verified.
Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization
TLDR
This paper presents an overview of recent work in decentralized optimization and surveys the state-of-theart algorithms and their analyses tailored to these different scenarios, highlighting the role of the network topology.
Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
TLDR
The error due to limits in communication resources decreases at a fast rate even in the case of non-strongly-convex objective functions, and the first optimal first-order decentralized algorithm called multi-step primal-dual (MSPD) and its corresponding optimal convergence rate are provided.
Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers
TLDR
It is argued that the alternating direction method of multipliers is well suited to distributed convex optimization, and in particular to large-scale problems arising in statistics, machine learning, and related areas.
The ADMM algorithm for distributed averaging: Convergence rates and optimal parameter selection
TLDR
This study derives the optimal step-size and over-relaxation parameter that minimizes the convergence time of two ADMM-based algorithms for distributed averaging and optimize the edge-weights of the communication graph to improve the convergence speed.
Distributed Estimation and Control of Algebraic Connectivity Over Random Graphs
TLDR
Using results from stochastic approximation theory, it is proved that the proposed method converges almost surely (a.s.) to the desired value of connectivity even in the presence of imperfect communication scenarios.
...
...