• Corpus ID: 222310239

A first look into the carbon footprint of federated learning

@article{Qiu2020AFL,
  title={A first look into the carbon footprint of federated learning},
  author={Xinchi Qiu and Titouan Parcollet and Daniel J. Beutel and Taner Topal and Akhil Mathur and Nicholas D. Lane},
  journal={ArXiv},
  year={2020},
  volume={abs/2102.07627}
}
Despite impressive results, deep learning-based technologies also raise severe privacy and environmental concerns induced by the training procedure often conducted in data centers. In response, alternatives to centralized training such as Federated Learning (FL) have emerged. Perhaps unexpectedly, FL in particular is starting to be deployed at a global scale by companies that must adhere to new legal demands and policies originating from governments and the civil society for privacy protection… 

Figures and Tables from this paper

FedSyn: Synthetic Data Generation using Federated Learning
TLDR
The proposed technique in this paper leverages federated machine learning and generative adversarial network as neural network architecture for synthetic data generation and can be extended to many machine learning problem classes in finance, health, governance, technology and many more.
FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
TLDR
This work introduces Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in deep neural networks (DNNs) and enables the extraction of lower footprint submodels without the need of retraining, and shows that for linear maps, this technique is equivalent to SVD.
Towards Quantifying the Carbon Emissions of Differentially Private Machine Learning
TLDR
This paper investigates the impact of differential privacy on learning algorithms in terms of their carbon footprint due to either longer run-times or failed experiments and provides guidance on choosing the noise levels which can strike a balance between desired privacy levels and reduced carbon emissions.
Gradient Masked Averaging for Federated Learning
TLDR
This work proposes a gradient masked averaging approach for federated learning as an alternative to the standard averaging of client updates, which can be adapted as a drop-in replacement in most existing federated algorithms.
Federated Self-supervised Speech Representations: Are We There Yet?
TLDR
This paper provides a first-of-its-kind systematic study of the feasibility and complexities for training speech SSL models under FL scenarios from the perspective of algorithms, hardware, and systems limits and suggests hybrid SSL and FL speech systems will not be viable until 2027.
DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning
TLDR
DAdaQuant consistently improves client→server compression, outperforming the strongest non-adaptive baselines by up to 2.8×, and is combined into DAdaquant, the doubly- Adaptive quantization algorithm.
Impossibility of Collective Intelligence
TLDR
This work shows that it is theoretically impossible to design a rational learning algorithm that has the ability to successfully learn across heterogeneous environments, which it decoratively call collective intelligence (CI) by representing learning algorithms as choice correspondences over a hypothesis space.
Energy Minimization for Federated Asynchronous Learning on Battery-Powered Mobile Devices via Application Co-running
TLDR
An online optimization framework is designed and implemented by connecting asynchronous execution of federated training with application co-running to minimize energy consumption on battery-powered mobile devices and can save over 60% energy with 3 times faster convergence speed.
Sustainable AI: Environmental Implications, Challenges and Opportunities
TLDR
The carbon footprint of AI computing is characterized by examining the model development cycle across industry-scale machine learning use cases and, at the same time, considering the life cycle of system hardware.
End-to-End Speech Recognition from Federated Acoustic Models
TLDR
This paper presents the first empirical study on attention-based sequence-to-sequence Endto-End ASR model with three aggregation weighting strategies – standard FedAvg, loss-based aggregation and a novel word error rate (WER)-based aggregation, compared in two realistic FL scenarios.
...
...

References

SHOWING 1-10 OF 54 REFERENCES
Advances and Open Problems in Federated Learning
TLDR
Motivated by the explosive growth in FL research, this paper discusses recent advances and presents an extensive collection of open problems and challenges.
Flower: A Friendly Federated Learning Research Framework
TLDR
Flower is presented, a FL framework which is both agnostic towards heterogeneous client environments and also scales to a large number of clients, including mobile and embedded devices and describes the design goals and implementation considerations of Flower.
Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models
TLDR
This work proposes that energy and carbon footprint of model development and training is reported alongside performance metrics using tools like Carbontracker, and hopes this will promote responsible computing in ML and encourage research into energy-efficient deep neural networks.
Federated Learning in Mobile Edge Networks: A Comprehensive Survey
TLDR
In a large-scale and complex mobile edge network, heterogeneous devices with varying constraints are involved, this raises challenges of communication costs, resource allocation, and privacy and security in the implementation of FL at scale.
Federated Optimization in Heterogeneous Networks
TLDR
This work introduces a framework, FedProx, to tackle heterogeneity in federated networks, and provides convergence guarantees for this framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work.
Adaptive Federated Optimization
TLDR
This work proposes federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyzes their convergence in the presence of heterogeneous data for general nonconvex settings to highlight the interplay between client heterogeneity and communication efficiency.
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
TLDR
A framework is introduced that makes accounting easier by providing a simple interface for tracking realtime energy consumption and carbon emissions, as well as generating standardized online appendices, and creates a leaderboard for energy efficient reinforcement learning algorithms to incentivize responsible research.
Federated Optimization: Distributed Optimization Beyond the Datacenter
We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are distributed (unevenly) over an extremely large
Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
TLDR
This work proposes a way to synthesize datasets with a continuous range of identicalness and provide performance measures for the Federated Averaging algorithm, and shows that performance degrades as distributions differ more, and proposes a mitigation strategy via server momentum.
Quantifying the Carbon Emissions of Machine Learning
TLDR
This work presents their Machine Learning Emissions Calculator, a tool for the community to better understand the environmental impact of training ML models and concrete actions that individual practitioners and organizations can take to mitigate their carbon emissions.
...
...