FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning

@article{Xiong2022FedDMID,
  title={FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning},
  author={Yuanhao Xiong and Ruochen Wang and Minhao Cheng and Felix Yu and Cho-Jui Hsieh},
  journal={ArXiv},
  year={2022},
  volume={abs/2207.09653}
}
Federated learning (FL) has recently attracted increasing attention from academia and industry, with the ultimate goal of achieving collaborative training under privacy and communication constraints. Existing iterative model averaging based FL algorithms require a large number of communication rounds to obtain a well-performed model due to extremely unbalanced and non-i.i.d data partitioning among different clients. Thus, we propose FedDM to build the global training objective from multiple… 
[PDF] Semantic Reader
7 Citations
Background Citations
3
Methods Citations
3

7 Citations

Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory

It is shown that there exists a procedure to exactly calculate the gradient of the trajectory matching loss with constant GPU memory requirement, and the proposed memory-efficient trajectory matching method can easily scale to ImageNet-1K with ∼ 6 x memory reduction while introducing only ∼ 2 % runtime overhead than original MTT.

Dataset Distillation: A Comprehensive Review

A comprehensive review and summary for recent advances in DD and its application is presented and an overall algorithmic framework followed by all existing DD methods is proposed.

A Comprehensive Survey to Dataset Distillation

A holistic understanding of dataset distillation from multiple aspects, including distillation frameworks and algorithms, disentangled datasets distillation, performance comparison, and applications is provided.

Accelerating Dataset Distillation via Model Augmentation

This paper proposes two model augmentation techniques, i.e .

Backdoor Attacks Against Dataset Distillation

This study performs the first backdoor attack against the models trained on the data distilled by dataset distillation models in the image domain, and proposes two types of backdoor attacks, namely N AIVE A TTACK and D OOR P ING.

A Review of Dataset Distillation for Deep Learning

This paper first elaborate on several dataset distillation methods for producing distilled datasets, and provides a summary of the datasets distillation-based solutions to deep learning tasks that have been presented in the major machine learning conferences in recent years.

Data Distillation: A Survey

This survey presents a formal framework for data distillation, along with providing a detailed taxonomy of existing approaches, and covers datadistillation approaches for different data modalities, namely images, graphs, and user-item interactions (recommender systems), while also identifying current challenges and future research directions.

References

SHOWING 1-10 OF 51 REFERENCES

Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data

Sparse ternary compression (STC) is proposed, a new compression framework that is specifically designed to meet the requirements of the federated learning environment and advocate for a paradigm shift in federated optimization toward high-frequency low-bitwidth communication, in particular in the bandwidth-constrained learning environments.

Communication-efficient federated learning

A communication-efficient FL framework is proposed to jointly improve the FL convergence time and the training loss, and a probabilistic device selection scheme is designed such that the devices that can significantly improve the convergence speed and training loss have higher probabilities of being selected for ML model transmission.

FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization

FedPAQ is presented, a communication-efficient Federated Learning method with Periodic Averaging and Quantization that achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by the method.

Communication-Efficient Federated Deep Learning With Layerwise Asynchronous Model Update and Temporally Weighted Aggregation

The results demonstrate that the proposed asynchronous federated deep learning outperforms the baseline algorithm both in terms of communication cost and model accuracy.

Federated Learning: Strategies for Improving Communication Efficiency

Two ways to reduce the uplink communication costs are proposed: structured updates, where the user directly learns an update from a restricted space parametrized using a smaller number of variables, e.g. either low-rank or a random mask; and sketched updates, which learn a full model update and then compress it using a combination of quantization, random rotations, and subsampling.

SCAFFOLD: Stochastic Controlled Averaging for Federated Learning

This work obtains tight convergence rates for FedAvg and proves that it suffers from `client-drift' when the data is heterogeneous (non-iid), resulting in unstable and slow convergence, and proposes a new algorithm (SCAFFOLD) which uses control variates (variance reduction) to correct for the ` client-drifts' in its local updates.

Distilled One-Shot Federated Learning

The proposed Distilled One-Shot Federated Learning, which reduces the number of communication rounds required to train a performant model to only one, and represents a new direction orthogonal to previous work, towards weight-less and gradient-less federated learning.

Federated Learning with Matched Averaging

This work proposes Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs and indicates that FedMA outperforms popular state-of-the-art federatedLearning algorithms on deep CNN and L STM architectures trained on real world datasets, while improving the communication efficiency.

Federated Optimization in Heterogeneous Networks

This work introduces a framework, FedProx, to tackle heterogeneity in federated networks, and provides convergence guarantees for this framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work.

A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection

A comprehensive review on federated learning systems is conducted and a thorough categorization is provided according to six different aspects, including data distribution, machine learning model, privacy mechanism, communication architecture, scale of federation and motivation of federation.
...