Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity

@article{Maeng2022TowardsFF,
  title={Towards Fair Federated Recommendation Learning: Characterizing the Inter-Dependence of System and Data Heterogeneity},
  author={Kiwan Maeng and Haiyu Lu and Luca Melis and John Nguyen and Michael G. Rabbat and Carole-Jean Wu},
  journal={Proceedings of the 16th ACM Conference on Recommender Systems},
  year={2022}
}
Federated learning (FL) is an effective mechanism for data privacy in recommender systems that runs machine learning model training on-device. While prior FL optimizations tackled the data and system heterogeneity challenges, they assume the two are independent of each other. This fundamental assumption is not reflective of real-world, large-scale recommender systems — data and system heterogeneity are tightly intertwined. This paper takes a data-driven approach to show the inter-dependence of… 

Figures and Tables from this paper

FEL: High Capacity Learning for Recommendation and Ranking via Federated Ensemble Learning

TLDR
Federated Ensemble Learning (FEL) is proposed as a solution to tackle the large memory requirement of deep learning ranking and recommendation tasks and leads to 0.43–2.31% model quality improvement over traditional on-device federated learning.

References

SHOWING 1-10 OF 79 REFERENCES

Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data

TLDR
The first empirical study to characterize the impacts of heterogeneity in Federated learning and build a heterogeneity-aware FL platform that complies with the standard FL protocol but with heterogeneity in consideration, which suggests that FL algorithm designers consider necessary heterogeneity during the evaluation.

Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer

TLDR
This work proposes a personalized FL framework, PERFED-CKT, where clients can use heterogeneous model architectures and do not directly communicate their model parameters and theoretically and empirically shows the convergence and generalization properties of this framework.

AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learning

TLDR
AutoFL is proposed by tailor-designing a reinforcement learning algorithm that learns and determines which K participant devices and per-device execution targets for each FL model aggregation round in the presence of stochastic runtime variance, system and data heterogeneity, and achieves 3.6 times faster model convergence time and 4.2 times higher energy efficiency.

A Field Guide to Federated Optimization

TLDR
This paper provides recommendations and guidelines on formulating, designing, evaluating and analyzing federated optimization algorithms through concrete examples and practical implementation, with a focus on conducting effective simulations to infer real-world performance.

Ensemble Distillation for Robust Model Fusion in Federated Learning

TLDR
This work proposes ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients, which allows flexible aggregation over heterogeneous client models that can differ e.g. in size, numerical precision or structure.

Billion-scale federated learning on mobile clients: a submodel design with tunable privacy

TLDR
A secure federated submodel learning scheme coupled with a private set union protocol as a cornerstone is designed, which features the properties of randomized response, secure aggregation, and Bloom filter, and endows each client with customized plausible deniability against the position of its desired submodel, thereby protecting private data.

Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning

TLDR
Adaptive Federated Dropout (AFD) is proposed and studied, a novel technique to reduce the communication costs associated with federated learning that optimizes both server-client communications and computation costs by allowing clients to train locally on a selected subset of the global model.

Oort: Efficient Federated Learning via Guided Participant Selection

TLDR
Oort improves time-to-accuracy performance in model training, and prioritizes the use of those clients who have both data that offers the greatest utility in improving model accuracy and the capability to run training quickly, to enable FL developers to interpret their results in model testing.

Adaptive Dynamic Pruning for Non-IID Federated Learning

TLDR
An adaptive pruning scheme for edge devices in an FL system, which applies datasetaware dynamic pruning for inference acceleration on Non-IID datasets and accelerates inference by 2× (50% FLOPs reduction) while maintaining the model’s quality on edge devices is presented.

The Architectural Implications of Facebook's DNN-Based Personalized Recommendation

TLDR
A set of real-world, production-scale DNNs for personalized recommendation coupled with relevant performance metrics for evaluation are presented and in-depth analysis is conducted that underpins future system design and optimization for at-scale recommendation.
...