# Private Ad Modeling with DP-SGD

@article{Denison2022PrivateAM,
author={Carson E. Denison and Badih Ghazi and Pritish Kamath and Ravi Kumar and Pasin Manurangsi and Krishnagiri Narra and Amer Sinha and Avinash V. Varadarajan and Chiyuan Zhang},
journal={ArXiv},
year={2022},
volume={abs/2211.11896}
}
• Published 21 November 2022
• Computer Science
• ArXiv
A well-known algorithm in privacy-preserving ML is differentially private stochastic gradient descent (DP-SGD). While this algorithm has been evaluated on text and image data, it has not been previously applied to ads data, which are noto-rious for their high class imbalance and sparse gradient updates. In this work we apply DP-SGD to several ad modeling tasks including predicting click-through rates, conversion rates, and number of conversion events, and evaluate their privacy-utility trade…

## References

SHOWING 1-10 OF 27 REFERENCES

• Computer Science
ArXiv
• 2019
AdaCliP is proposed, a theoretically motivated differentially private SGD algorithm that provably adds less noise compared to the previous methods, by using coordinate-wise adaptive clipping of the gradient.
• Computer Science
ArXiv
• 2022
It is demonstrated that DP-SGD on over-parameterized models can perform signiﬁcantly better than previously thought and is believed to be a step towards closing the accuracy gap between private and non-private image classi-cation benchmarks.
• Computer Science
CCS
• 2016
This work develops new algorithmic techniques for learning and a refined analysis of privacy costs within the framework of differential privacy, and demonstrates that deep neural networks can be trained with non-convex objectives, under a modest privacy budget, and at a manageable cost in software complexity, training efficiency, and model quality.
• Computer Science
ArXiv
• 2022
Initial lessons from this effort to investigate how to train differential privacy training at scale are shared, showing approaches that help make DP training faster, as well as model types and settings of the training process that tend to work better in the DP setting.
• Computer Science
Proc. Priv. Enhancing Technol.
• 2021
New methods for per-example gradient clipping that are compatible with auto-differeniation and provide better GPU utilization are derived by analyzing the back-propagation equations of Renyi Differential Privacy.
• Computer Science
FINDINGS
• 2022
By using recent advances in JAX and XLA, this work can train models with DP that do not suffer a large drop in pre-training utility, nor in training speed, and can still be fine-tuned to high accuracies on downstream tasks (e.g. GLUE).
• Computer Science
NeurIPS
• 2021
This paper proposes an algorithmic solution which works for any network in a black-box manner and trains a Recurrent Neural Network to achieve good privacy-vs-accuracy tradeoff, while being signiﬁcantly faster than DP-SGD and with a similar memory footprint as non-private SGD.
• Computer Science
AISTATS
• 2020
A numerical accountant can be applied to the subsampled multidimensional Gaussian mechanism which underlies the popular DP stochastic gradient descent and gives the exact $(\varepsilon,\delta)$-values.
• Computer Science
CCS
• 2018
With privacy buckets, a numerical and widely applicable method for capturing the privacy loss of differentially private mechanisms under composition, which is called privacy buckets is presented and it is shown that for concrete sequences of mechanisms tighter bounds can be derived by taking the mechanisms' structure into account.
• Computer Science
NeurIPS
• 2021
It is shown that adaptively setting the clipping norm applied to each user's update, based on a differentially private estimate of a target quantile of the distribution of unclipped norms, is sufficient to remove the need for such extensive parameter tuning.