• Corpus ID: 224706936

Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization

@article{Subramani2020EnablingFD,
  title={Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorization},
  author={Pranav Subramani and Nicholas Vadivelu and Gautam Kamath},
  journal={ArXiv},
  year={2020},
  volume={abs/2010.09063}
}
A common pain point in differentially private machine learning is the significant runtime overhead incurred when executing Differentially Private Stochastic Gradient Descent (DPSGD), which may be as large as two orders of magnitude. We thoroughly demonstrate that by exploiting powerful language primitives, including vectorization, just-in-time compilation, and static graph optimization, one can dramatically reduce these overheads, in many cases nearly matching the best non-private running times… 
Fast and Memory Efficient Differentially Private-SGD via JL Projections
TLDR
This paper proposes an algorithmic solution which works for any network in a black-box manner and trains a Recurrent Neural Network to achieve good privacy-vs-accuracy tradeoff, while being significantly faster than DP-SGD and with a similar memory footprint as non-private SGD.
Large-Scale Differentially Private BERT
TLDR
This work studies the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD), and shows that scaling up the batch size to millions improves the utility of the DP- SGD step for BERT and enhances its efficiency by using an increasing batch size schedule.
NanoBatch DPSGD: Exploring Differentially Private learning on ImageNet with low batch sizes on the IPU
TLDR
It is argued that low batch sizes using group normalization on ResNet-50 can yield high accuracy and privacy on Graphcore IPUs and enables DPSGD training of Res net-50 on ImageNet in just 6 hours (100 epochs) on an IPU-POD16 system.
Differentially Private Fine-tuning of Language Models
We give simpler, sparser, and faster algorithms for differentially private fine-tuning of large-scale pre-trained language models, which achieve the state-of-the-art privacy versus utility tradeoffs
DP-FP: Differentially Private Forward Propagation for Large Models
TLDR
This work shows how to mitigate the performance drop by replacing the DPSGD with a novel DP Forward-Propagation (DP-FP) followed by an off-the-shelf non-DP optimizer, which outperforms cutting-edge DP-SGD while retaining the same level of privacy, and it approaches non-private baselines and significantly outperforms state-of- the-art DP- SGD variants.
Opacus: User-Friendly Differential Privacy Library in PyTorch
TLDR
Opacus, a free, open-source PyTorch library for training deep learning models with differential privacy, is introduced and the principles that drove its implementation and unique features are detailed, and its performance against other frameworks for differential privacy in ML is evaluated.
d3p - A Python Package for Differentially-Private Probabilistic Programming
We present d3p, a software package designed to help fielding runtime efficient widely-applicable Bayesian inference under differential privacy guarantees. d3p achieves general applicability to a wide
Medical imaging deep learning with differential privacy
The successful training of deep learning models for diagnostic deployment in medical imaging applications requires large volumes of data. Such data cannot be procured without consideration for
DPNAS: Neural Architecture Search for Deep Learning with Differential Privacy
TLDR
This paper proposes the very first framework that employs neural architecture search to automatic model design for private deep learning, dubbed as DPNAS, delicately design a novel search space and proposes a DP-aware method for training candidate models, and empirically certify the effectiveness of the proposed framework.
Adversarial Attacks Against Deep Generative Models on Data: A Survey
TLDR
This comprehensive and specialized survey on the security and privacy preservation of GANs and VAEs focuses on the inner connection between attacks and model architectures and, more specifically, on five components of deep generative models.
...
1
2
...

References

SHOWING 1-10 OF 78 REFERENCES
Fast and Memory Efficient Differentially Private-SGD via JL Projections
TLDR
This paper proposes an algorithmic solution which works for any network in a black-box manner and trains a Recurrent Neural Network to achieve good privacy-vs-accuracy tradeoff, while being significantly faster than DP-SGD and with a similar memory footprint as non-private SGD.
Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clipping
TLDR
New methods for per-example gradient clipping that are compatible with auto-differeniation and provide better GPU utilization are derived by analyzing the back-propagation equations of Renyi Differential Privacy.
PyTorch: An Imperative Style, High-Performance Deep Learning Library
TLDR
This paper details the principles that drove the implementation of PyTorch and how they are reflected in its architecture, and explains how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance.
BackPACK: Packing more into backprop
TLDR
BackPACK is introduced, an efficient framework built on top of PyTorch that extends the backpropagation algorithm to extract additional information from first-and second-order derivatives to address the problem of automatic differentiation frameworks not supporting other quantities such as the variance of the mini-batch gradients.
Large-Scale Differentially Private BERT
TLDR
This work studies the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD), and shows that scaling up the batch size to millions improves the utility of the DP- SGD step for BERT and enhances its efficiency by using an increasing batch size schedule.
Compiling machine learning programs via high-level tracing
TLDR
JAX is described, a domain-specific tracing JIT compiler for generating high-performance accelerator code from pure Python and Numpy machine learning programs that is capable of scaling to multi-core Cloud TPUs and easily programmable and highly performant ML system.
Opacus: User-Friendly Differential Privacy Library in PyTorch
TLDR
Opacus, a free, open-source PyTorch library for training deep learning models with differential privacy, is introduced and the principles that drove its implementation and unique features are detailed, and its performance against other frameworks for differential privacy in ML is evaluated.
TensorFlow: A system for large-scale machine learning
TLDR
The TensorFlow dataflow model is described and the compelling performance that Tensor Flow achieves for several real-world applications is demonstrated.
Towards Practical Differentially Private Convex Optimization
TLDR
Approximate Minima Perturbation is presented, a novel algorithm that can leverage any off-the-shelf optimizer and can be employed without any hyperparameter tuning, thus making it an attractive technique for practical deployment.
Auto-Vectorizing TensorFlow Graphs: Jacobians, Auto-Batching And Beyond
TLDR
A new statically vectorized parallel-for abstraction is provided on top of TensorFlow, and used for applications ranging from auto-batching and per-example gradients, to jacobian computation, optimized map functions and input pipeline optimization.
...
1
2
3
4
5
...