Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning

@article{Elgabli2020QGADMMQG,
  title={Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning},
  author={Anis Elgabli and Jihong Park and A. S. Bedi and Mehdi Bennis and Vaneet Aggarwal},
  journal={ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2020},
  pages={8876-8880}
}
  • Anis Elgabli, Jihong Park, +2 authors V. Aggarwal
  • Published 23 October 2019
  • Computer Science, Mathematics
  • ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
In this paper, we propose a communication-efficient decen-tralized machine learning (ML) algorithm, coined quantized group ADMM (Q-GADMM). Every worker in Q-GADMM communicates only with two neighbors, and updates its model via the group alternating direct method of multiplier (GADMM), thereby ensuring fast convergence while reducing the number of communication rounds. Furthermore, each worker quantizes its model updates before transmissions, thereby decreasing the communication payload sizes… Expand
Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning
TLDR
A novel stochastic quantization method is developed to adaptively adjust model quantization levels and their probabilities, while proving the convergence of Q-GADMM for convex objective functions. Expand
Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM
TLDR
Numerical simulations corroborate that CQ-GGADMM exhibits higher communication efficiency in terms of the number of communication rounds and transmit energy consumption without compromising the accuracy and convergence speed, compared to the benchmark schemes based on decentralized ADMM without censoring, quantization, and/or the worker grouping method of GADMM. Expand
Generalized ADMM in Distributed Learning via Variational Inequality
TLDR
The aim is to deploy variational inequality (VI) to attain an unified view of ADMM variations through the simulation results to demonstrate how more general definitions of consensus parameters and introducing the uncertain parameters in distribute approach can help to get the better results in learning processes. Expand
Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications
TLDR
This article aims to provide a holistic overview of relevant communication and ML principles and present communication-efficient and distributed learning frameworks with selected use cases. Expand
Federated Learning and Control at the Wireless Network Edge
  • M. Bennis
  • Computer Science
  • GetMobile Mob. Comput. Commun.
  • 2020
TLDR
The fundamentals of ultra-reliable and low-latency communication (URLLC), one of the major tenets of the 5G revolution, are yet to be fully understood and warrants a departure from average-based system design toward a clean slate design centered on tail, risk, and scale. Expand
Wireless Edge Machine Learning: Resource Allocation and Trade-Offs
TLDR
A dynamic resource allocation framework for EML based on stochastic Lyapunov optimization for dynamic training and inference of machine learning tasks at the edge of the wireless network, with the goal of exploring the trade-off between energy, delay and learning accuracy. Expand
From Federated Learning to Fog Learning: Towards Large-Scale Distributed Machine Learning in Heterogeneous Wireless Networks
TLDR
Fog learning is advocated, which will intelligently distribute ML model training across the fog, the continuum of nodes from edge devices to cloud servers, and several open research directions for fog learning are discussed. Expand
6G White Paper on Edge Intelligence
TLDR
This white paper envision a transition from Internet of Things to Intelligent Internet of Intelligent Things and provide a roadmap for development of 6G Intelligent Edge, along with security, privacy, pricing, and end-user aspects. Expand
Link Adjustment for Assembly Deviation Control of Extendible Support Structures via Sparse Optimization
TLDR
A novel method to quantitatively conduct dimensional adjustment of links for extendible support structures of synthetic aperture radar antennas and can not only achieve satisfactory performance in prediction but also significantly improve the assembly efficiency is proposed. Expand
Beyond Private 5G  Networks: Applications, Architectures, Operator Models and Technological Enablers
TLDR
This paper applies the proposed operator models to different network architecture options and to a selection of relevant use cases offering mixed private-public network operator governance and ownership to present a novel approach to operator models, specifically targeting 5G and beyond private networks. Expand
...
1
2
...

References

SHOWING 1-10 OF 50 REFERENCES
GADMM: Fast and Communication Efficient Framework for Distributed Machine Learning
TLDR
It is proved that GADMM converges to the optimal solution for convex loss functions, and numerically show that it converges faster and more communication-efficient than the state-of-the-art communication- efficient algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in linear and logistic regression tasks on synthetic and real datasets. Expand
Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients
TLDR
A novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication by first quantizing the computed gradients, and then skipping less informative quantized gradient communications by reusing outdated gradients. Expand
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
TLDR
This paper considers a distributed communication efficient momentum SGD method and proves its linear speedup property, filling the gap in the study of distributed SGD variants with reduced communication. Expand
LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning
TLDR
A new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation is presented, justifying the acronym LAG used henceforth. Expand
Federated Learning: Strategies for Improving Communication Efficiency
TLDR
Two ways to reduce the uplink communication costs are proposed: structured updates, where the user directly learns an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, which learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling. Expand
Communication-Efficient Distributed Dual Coordinate Ascent
TLDR
A communication-efficient framework that uses local computation in a primal-dual setting to dramatically reduce the amount of necessary communication is proposed, and a strong convergence rate analysis is provided for this class of algorithms. Expand
Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
TLDR
Federated distillation (FD) is proposed, a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Expand
A Method to Improve Consensus Averaging using Quantized ADMM
TLDR
An algorithm is proposed which leads to perfect consensus in an asymptotic sense, without the need to increase the per-exchange communication rate of the network. Expand
Group-Based Alternating Direction Method of Multipliers for Distributed Linear Classification
TLDR
Compared with disADMM and stochastic dual coordinate ascent with alternating direction method of multipliers-ADMM, for distributed classification, GADMM is able to reduce the number of outer iterations, which leads to faster convergence speed and better global consensus. Expand
Quantized consensus ADMM for multi-agent distributed optimization
  • Shengyu Zhu, Mingyi Hong, B. Chen
  • Mathematics, Computer Science
  • 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • 2016
TLDR
A quantized distributed algorithm based on the alternating direction method of multipliers (ADMM) which converges to a consensus within log1+η Ω iterations and obtains a tight upper bound on the consensus error which does not depend on the size of the network. Expand
...
1
2
3
4
5
...