Estimation, Optimization, and Parallelism when Data is Sparse or Highly Varying

Abstract

<lb>We study stochastic optimization problems when the data is sparse, which is in a sense<lb>dual to the current understanding of high-dimensional statistical learning and optimization.<lb>We highlight both the difficulties—in terms of increased sample complexity that sparse data<lb>necessitates—and the potential benefits, in terms of allowing parallelism and asynchrony in the<lb>design of algorithms. Concretely, we derive matching upper and lower bounds on the minimax<lb>rate for optimization and learning with sparse data, and we exhibit algorithms achieving these<lb>rates. We also show how leveraging sparsity leads to (still minimax optimal) parallel and<lb>asynchronous algorithms, providing experimental evidence complementing our theoretical results<lb>on several medium to large-scale learning tasks.

3 Figures and Tables

Cite this paper

@inproceedings{Duchi2013EstimationOA, title={Estimation, Optimization, and Parallelism when Data is Sparse or Highly Varying}, author={John C. Duchi and Michael I. Jordan and H. Brendan McMahan}, year={2013} }