• Corpus ID: 14999259

Federated Learning: Strategies for Improving Communication Efficiency

@article{Konecn2016FederatedLS,
  title={Federated Learning: Strategies for Improving Communication Efficiency},
  author={Jakub Konecn{\'y} and H. B. McMahan and Felix X. Yu and Peter Richt{\'a}rik and Ananda Theertha Suresh and Dave Bacon},
  journal={ArXiv},
  year={2016},
  volume={abs/1610.05492}
}
Federated Learning is a machine learning setting where the goal is to train a high-quality centralized model while training data remains distributed over a large number of clients each with unreliable and relatively slow network connections. [] Key Method In this paper, we propose two ways to reduce the uplink communication costs: structured updates, where we directly learn an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched…

Figures from this paper

FEDZIP: A Compression Framework for Communication-Efficient Federated Learning
TLDR
This work proposes a novel framework, FedZip, that significantly decreases the size of updates while transferring weights from the deep learning model between clients and their servers, and outperforms state-of-the-art compression frameworks and reaches compression rates up to 1085×.
Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning
TLDR
Adaptive Federated Dropout (AFD) is proposed and studied, a novel technique to reduce the communication costs associated with federated learning that optimizes both server-client communications and computation costs by allowing clients to train locally on a selected subset of the global model.
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
TLDR
FedPAQ is presented, a communication-efficient Federated Learning method with Periodic Averaging and Quantization that achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by the method.
Communication-Efficient Federated Learning via Optimal Client Sampling
TLDR
This work proposes a novel, simple and efficient way of updating the central model in communication-constrained settings based on collecting models from clients with informative updates and estimating local updates that were not communicated, and modeling the progression of model's weights by an Ornstein-Uhlenbeck process.
Federated Learning with Quantization Constraints
TLDR
This work identifies the unique characteristics associated with conveying trained models over rate-constrained channels, and characterize a suitable quantization scheme for such setups, and shows that combining universal vector quantization methods with FL yields a decentralized training system, which is both efficient and feasible.
Federated Learning With Quantized Global Model Updates
TLDR
A lossy FL (LFL) algorithm, in which both the global model and the local model updates are quantized before being transmitted, and it is shown that the quantization of theglobal model can actually improve the performance for non-iid data distributions.
Intrinisic Gradient Compression for Federated Learning
TLDR
This paper uses a correspondence between the notion of intrinsic dimension and gradient compressibility to derive a family of low-bandwidth optimization algorithms, which they are called intrinsic gradient compression algorithms.
Communication-Efficient Federated Learning With Binary Neural Networks
TLDR
A novel FL framework of training BNNs is introduced, where the clients only upload the binary parameters to the server, and a novel parameter updating scheme based on the Maximum Likelihood (ML) estimation that preserves the performance of the BNN even without the availability of aggregated real-valued auxiliary parameters.
Hierarchical Quantized Federated Learning: Convergence Analysis and System Design
TLDR
This paper considers a Hierarchical Quantized Federated Learning (HQFL) system with one cloud server, several edge servers and many clients, adopting a communication-efficient training algorithm, Hier-Local-QSGD, and finds that given a latency budget for the whole training process, there is an optimal parameter choice.
Federated Learning with Heterogeneous Quantization
TLDR
This paper proposes FEDHQ: Federated Learning with Heterogeneous Quantization, a federated learning system with performance advantages over conventional FEDAVG with standard equal weights and a heuristic scheme which assigns weights linearly proportional to the clients’ quantization precision.
...
...

References

SHOWING 1-10 OF 30 REFERENCES
Federated Optimization: Distributed Machine Learning for On-Device Intelligence
We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are unevenly distributed over an extremely large number
Federated Learning of Deep Networks using Model Averaging
TLDR
This work presents a practical method for the federated learning of deep networks that proves robust to the unbalanced and non-IID data distributions that naturally arise, and allows high-quality models to be trained in relatively few rounds of communication.
Communication-Efficient Learning of Deep Networks from Decentralized Data
TLDR
This work presents a practical method for the federated learning of deep networks based on iterative model averaging, and conducts an extensive empirical evaluation, considering five different model architectures and four datasets.
Federated Optimization: Distributed Optimization Beyond the Datacenter
We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are distributed (unevenly) over an extremely large
Large Scale Distributed Deep Networks
TLDR
This paper considers the problem of training a deep network with billions of parameters using tens of thousands of CPU cores and develops two algorithms for large-scale distributed training, Downpour SGD and Sandblaster L-BFGS, which increase the scale and speed of deep network training.
Revisiting Distributed Synchronous SGD
TLDR
It is demonstrated that a third approach, synchronous optimization with backup workers, can avoid asynchronous noise while mitigating for the worst stragglers and is empirically validated and shown to converge faster and to better test accuracies.
Quantized incremental algorithms for distributed optimization
  • M. Rabbat, R. Nowak
  • Computer Science
    IEEE Journal on Selected Areas in Communications
  • 2005
TLDR
The main conclusion is that as the number of sensors in the network grows, in-network processing will always use less energy than a centralized algorithm, while maintaining a desired level of accuracy.
Adding vs. Averaging in Distributed Primal-Dual Optimization
TLDR
A novel generalization of the recent communication-efficient primal-dual framework (COCOA) for distributed optimization, which allows for additive combination of local updates to the global parameters at each iteration, whereas previous schemes with convergence guarantees only allow conservative averaging.
AIDE: Fast and Communication Efficient Distributed Optimization
TLDR
An accelerated variant of the DANE algorithm, called AIDE, is proposed that not only matches the communication lower bounds but can also be implemented using a purely first-order oracle.
Distributed optimization with arbitrary local solvers
TLDR
This work presents a framework for distributed optimization that both allows the flexibility of arbitrary solvers to be used on each (single) machine locally and yet maintains competitive performance against other state-of-the-art special-purpose distributed methods.
...
...