Corpus ID: 221006240

Federated Transfer Learning with Dynamic Gradient Aggregation

@article{Dimitriadis2020FederatedTL,
  title={Federated Transfer Learning with Dynamic Gradient Aggregation},
  author={Dimitrios Dimitriadis and Ken'ichi Kumatani and Robert Gmyr and Yashesh Gaur and Sefik Emre Eskimez},
  journal={ArXiv},
  year={2020},
  volume={abs/2008.02452}
}
In this paper, a Federated Learning (FL) simulation platform is introduced. The target scenario is Acoustic Model training based on this platform. To our knowledge, this is the first attempt to apply FL techniques to Speech Recognition tasks due to the inherent complexity. The proposed FL platform can support different tasks based on the adopted modular design. As part of the platform, a novel hierarchical optimization scheme and two gradient aggregation methods are proposed, leading to almost… Expand
Dynamic Gradient Aggregation for Federated Domain Adaptation
TLDR
The proposed FL system outperforms the baseline systems in both convergence speed and overall model performance, and investigates the effect of the FL algorithm in supervised and unsupervised Speech Recognition (SR) scenarios. Expand
Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework
TLDR
A framework by which the degree of non-IID-ness can be varied is proposed, consequently illustrating a trade-off between model quality and the computational cost of federated training, which is captured through a novel metric. Expand
Privacy attacks for automatic speech recognition acoustic models in a federated learning framework
TLDR
This paper proposes an approach to analyze information in neural network AMs based on a neural network footprint on the so-called Indicator dataset and develops two attack models that aim to infer speaker identity from the updated personalized models without access to the actual users’ speech data. Expand
Federated Transfer Learning: concept and applications
TLDR
This work provides a comprehensive survey of the existing works on federated transfer learning and studies the background of FTL and its different existing applications, and analyses FTL from privacy and machine learning perspective. Expand
Sequence-Level Self-Learning with Multiple Hypotheses
TLDR
New self-learning techniques with an attention-based sequence-to-sequence (seq2seq) model for automatic speech recognition (ASR) using the multi-task learning (MTL) framework where the n-th best ASR hypothesis is used as the label of each task. Expand

References

SHOWING 1-10 OF 38 REFERENCES
Online Learning to Sample
TLDR
This work shows that SGD can be used to learn the best possible sampling distribution of an importance sampling estimator, and shows that the sampling Distribution of an SGD algorithm can be estimated online by incrementally minimizing the variance of the gradient. Expand
Online Deep Learning: Learning Deep Neural Networks on the Fly
TLDR
A new ODL framework is presented that attempts to tackle the challenges by learning DNN models which dynamically adapt depth from a sequence of training data in an online learning setting by proposing a novel Hedge Backpropagation method for online updating the parameters of DNN effectively. Expand
Accelerating Stochastic Gradient Descent via Online Learning to Sample
TLDR
This work shows that SGD can be used to learn the best possible sampling distribution of an importance sampling estimator, and shows that the sampling Distribution of a SGD algorithm can be estimated online by incrementally minimizing the variance of the gradient. Expand
Federated Learning for Keyword Spotting
TLDR
An extensive empirical study of the federated averaging algorithm for the "Hey Snips" wake word based on a crowdsourced dataset that mimics a federation of wake word users shows that using an adaptive averaging strategy inspired from Adam highly reduces the number of communication rounds required to reach the target performance. Expand
Communication-Efficient Learning of Deep Networks from Decentralized Data
TLDR
This work presents a practical method for the federated learning of deep networks based on iterative model averaging, and conducts an extensive empirical evaluation, considering five different model architectures and four datasets. Expand
A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
  • Q. Li, Zeyi Wen, B. He
  • Computer Science, Mathematics
  • IEEE Transactions on Knowledge and Data Engineering
  • 2021
TLDR
A comprehensive review on federated learning systems is conducted and a thorough categorization is provided according to six different aspects, including data distribution, machine learning model, privacy mechanism, communication architecture, scale of federation and motivation of federation. Expand
Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering
  • Kai Chen, Qiang Huo
  • Computer Science
  • 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • 2016
TLDR
This work has trained successfully deep bidirectional long short-term memory (LSTM) recurrent neural networks and fully-connected feed-forward deep neural networks (DNNs) for large vocabulary continuous speech recognition on two benchmark tasks, namely 309-hour Switchboard-I and 1,860-hour "Switch-board+Fisher" task. Expand
Sequence-level self-learning with multi-task learning framework
TLDR
New self-learning techniques with an attention-based sequence-to-sequence (seq2seq) model for automatic speech recognition (ASR) using the multi-task learning (MTL) framework where then n-th best ASR hypothesis is used as the label of each task. Expand
Large-Scale Domain Adaptation via Teacher-Student Learning
TLDR
This work proposes an approach to domain adaptation that does not require transcriptions but instead uses a corpus of unlabeled parallel data, consisting of pairs of samples from the source domain of the well-trained model and the desired target domain, to perform adaptation. Expand
Asynchronous Decentralized Learning of a Neural Network
TLDR
This work exploits an asynchronous computing framework namely ARock to learn a deep neural network called self-size estimating feedforward neural network (SSFN) in a decentralized scenario and provides the centralized equivalent solution under certain technical assumptions, namely asynchronous decentralized SSFN (dSSFN). Expand
...
1
2
3
4
...