# Robust Estimation via Robust Gradient Estimation

@article{Prasad2018RobustEV,
title={Robust Estimation via Robust Gradient Estimation},
author={A. Prasad and Arun Sai Suggala and Sivaraman Balakrishnan and Pradeep Ravikumar},
journal={ArXiv},
year={2018},
volume={abs/1802.06485}
}
We provide a new computationally-efficient class of estimators for risk minimization. We show that these estimators are robust for general statistical models: in the classical Huber epsilon-contamination model and in heavy-tailed settings. Our workhorse is a novel robust variant of gradient descent, and we provide conditions under which our gradient descent variant provides accurate estimators in a general convex risk minimization problem. We provide specific consequences of our theory for… Expand
134 Citations

#### Figures, Tables, and Topics from this paper

A Unified Approach to Robust Mean Estimation
• Computer Science, Mathematics
• ArXiv
• 2019
In this paper, we develop connections between two seemingly disparate, but central, models in robust statistics: Huber's epsilon-contamination model and the heavy-tailed noise model. We provideExpand
A Robust Univariate Mean Estimator is All You Need
• Mathematics, Computer Science
• AISTATS
• 2020
This work designs statistically optimal estimators for flexible non-parametric distribution classes such as distributions with bounded-2k moments and symmetric distributions, and designs estimators which are optimal in both heavy-tailed and contaminated settings. Expand
High-Dimensional Robust Mean Estimation via Gradient Descent
• Yu Cheng
• Computer Science, Mathematics
• ICML
• 2020
This work establishes an intriguing connection between algorithmic high-dimensional robust statistics and non-convex optimization, which may have broader applications to other robust estimation tasks. Expand
Robust descent using smoothed multiplicative noise
This work proposes a novel robust gradient descent procedure which makes use of a smoothed multiplicative noise applied directly to observations before constructing a sum of soft-truncated gradient coordinates, and shows that the procedure has competitive theoretical guarantees. Expand
High Dimensional Robust Estimation of Sparse Models via Trimmed Hard Thresholding
• Computer Science, Mathematics
• ArXiv
• 2019
This work studies the problem of sparsity constrained M -estimation with arbitrary corruptions to both explanatory and response variables in the high-dimensional regime, and develops a highly efficient gradient-based optimization algorithm that is a robust variant of Iterative Hard Thresholding by using trimmed mean in gradient computations. Expand
Adaptive Hard Thresholding for Near-optimal Consistent Robust Regression
• Computer Science, Mathematics
• COLT
• 2019
A nearly linear time estimator which consistently estimates the true regression vector, even with $1-o(1)$ fraction of corruptions is provided, based on a novel variant of outlier removal via hard thresholding in which the threshold is chosen adaptively and crucially relies on randomness to escape bad fixed points of the non-convexhard thresholding operation. Expand
High Dimensional Robust Sparse Regression
We provide a novel – and to the best of our knowledge, the first – algorithm for high dimensional sparse regression with constant fraction of corruptions in explanatory and/or response variables. OurExpand
High Dimensional Robust Sparse Regression
We provide a novel – and to the best of our knowledge, the first – algorithm for high dimensional sparse regression with constant fraction of corruptions in explanatory and/or response variables. OurExpand
High Dimensional Robust Sparse Regression
• Computer Science, Mathematics
• AISTATS
• 2020
A filtering algorithm which consists of a novel randomized outlier removal technique for robust sparse mean estimation that may be of interest in its own right: the filtering algorithm is flexible enough to deal with unknown covariance. Expand
M L ] 1 J ul 2 01 9 A Unified Approach to Robust Mean Estimation
In this paper, we develop connections between two seemingly disparate, but central, models in robust statistics: Huber’s ǫ-contamination model and the heavy-tailed noise model. We provide conditionsExpand

#### References

SHOWING 1-10 OF 71 REFERENCES
Variance-based Regularization with Convex Objectives
• Computer Science, Mathematics
• NIPS
• 2017
An approach to risk minimization and stochastic optimization that provides a convex surrogate for variance, allowing near-optimal and computationally efficient trading between approximation and estimation error, and it is shown that this procedure comes with certificates of optimality. Expand
Computationally Efficient Robust Estimation of Sparse Functionals
• Computer Science, Mathematics
• ArXiv
• 2017
The theory identifies a unified set of deterministic conditions under which the algorithm guarantees accurate recovery of sparse functionals, and applies to many problems of considerable interest including sparse mean and covariance estimation; sparse linear regression; and sparse generalized linear models. Expand
ROBUST EMPIRICAL MEAN ESTIMATORS
• Mathematics
• 2011
We study robust estimators of the mean of a probability measure P, called robust empirical mean estimators. This elementary construction is then used to revisit a problem of aggregation and a problemExpand
Statistical consistency and asymptotic normality for high-dimensional robust M-estimators
This work establishes a form of local statistical consistency for the penalized regression estimators under fairly mild conditions on the error distribution, and analysis of the local curvature of the loss function has useful consequences for optimization when the robust regression function and/or regularizer is nonconvex and the objective function possesses stationary points outside the local region. Expand
Loss Minimization and Parameter Estimation with Heavy Tails
• Mathematics, Computer Science
• J. Mach. Learn. Res.
• 2016
The technique can be used for approximate minimization of smooth and strongly convex losses, and specifically for least squares linear regression and low-rank covariance matrix estimation with similar allowances on the noise and covariate distributions. Expand
Robust Sparse Estimation Tasks in High Dimensions
For both of these problems, the natural robust version of two classical sparse estimation problems, namely, sparse mean estimation and sparse PCA in the spiked covariance model, is studied, providing the first efficient algorithms that provide non-trivial error guarantees in the presence of noise. Expand
A General Decision Theory for Huber's $\epsilon$-Contamination Model
• Mathematics
• 2015
Today's data pose unprecedented challenges to statisticians. It may be incomplete, corrupted or exposed to some unknown source of contamination. We need new methods and theories to grapple with theseExpand
Robust machine learning by median-of-means: Theory and practice
• Mathematics, Computer Science
• 2017
New estimators for robust machine learning based on median-of-means (MOM) estimators of the mean of real valued random variables, which achieve optimal rates of convergence under minimal assumptions on the dataset and are easily computable in practice. Expand
High-dimensional regression with noisy and missing data: Provable guarantees with non-convexity
• Computer Science, Mathematics
• NIPS
• 2011
This work is able to both analyze the statistical error associated with any global optimum, and prove that a simple algorithm based on projected gradient descent will converge in polynomial time to a small neighborhood of the set of all global minimizers. Expand
Self Scaled Regularized Robust Regression
• Mathematics, Computer Science
• 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
• 2015
The main result shows that this approach is equivalent to a “self-scaled” ℓ1 regularized robust regression problem, where the cost function is automatically scaled, with scalings that depend on the a-priori information. Expand