# Low rank approximation and regression in input sparsity time

@inproceedings{Clarkson2013LowRA, title={Low rank approximation and regression in input sparsity time}, author={Kenneth L. Clarkson and David P. Woodruff}, booktitle={STOC '13}, year={2013} }

We design a new distribution over poly(r ε<sup>-1</sup>) x n matrices S so that for any fixed n x d matrix A of rank r, with probability at least 9/10, SAx<sub>2</sub> = (1 pm ε)Ax<sub>2</sub> simultaneously for all x ∈ R<sup>d</sup>. Such a matrix S is called a <i>subspace embedding</i>. Furthermore, SA can be computed in O(nnz(A)) + ~O(r<sup>2</sup>ε<sup>-2</sup>) time, where nnz(A) is the number of non-zero entries of A. This improves over all previous subspace embeddings, which required at…

## 224 Citations

Low-Rank PSD Approximation in Input-Sparsity Time

- Computer Science, MathematicsSODA
- 2017

This work gives algorithms for approximation by low-rank positive semidefinite (PSD) matrices, and shows that there are asymmetric input matrices that cannot have good symmetric column-selected approximations.

OSNAP: Faster Numerical Linear Algebra Algorithms via Sparser Subspace Embeddings

- Computer Science2013 IEEE 54th Annual Symposium on Foundations of Computer Science
- 2013

The main result is essentially a Bai-Yin type theorem in random matrix theory and is likely to be of independent interest: for any fixed U ∈ R<sup>n×d</sup> with orthonormal columns and random sparse Π, all singular values of ΠU lie in [1 - ε, 1 + ε] with good probability.

Approximation Algorithms for ࡁ0-Low Rank Approximation

- Computer ScienceArXiv
- 2017

This work gives the first algorithm with provable guarantees for the l0-Low Rank Approximation Problem for k > 1, even for bicriteria algorithms, and strengthens this for the well-studied case of binary matrices to obtain a (1+O(ψ))-approximation in sublinear time.

Tighter Low-rank Approximation via Sampling the Leveraged Element

- Computer ScienceSODA
- 2015

This work proposes a new randomized algorithm for computing a low-rank approximation to a given matrix that combines the best aspects of otherwise disparate current results, but with a dependence on the condition number κ = σ1/σr.

An homotopy method for lp regression provably beyond self-concordance and in input-sparsity time

- Mathematics, Computer ScienceSTOC
- 2018

It is proved that any symmetric self-concordant barrier on the ℓpn unit ball has self- Concordance parameter Ω(n), and a randomized algorithm solving such problems in input sparsity time is proposed, i.e., Õp(N + poly(d)) where N is the size of the input and d is the number of variables.

Input Sparsity Time Low-rank Approximation via Ridge Leverage Score Sampling

- Computer ScienceSODA
- 2017

We present a new algorithm for finding a near optimal low-rank approximation of a matrix $A$ in $O(nnz(A))$ time. Our method is based on a recursive sampling scheme for computing a representative…

D S ] 6 A pr 2 01 8 Tight Bounds for l p Oblivious Subspace Embeddings

- Computer Science, Mathematics
- 2018

This paper gives nearly optimal tradeoffs for lp oblivious subspace embeddings with an expected 1+ε number of non-zero entries per column for arbitrarily small ε > 0, and gives the first oblivious sub space embeds for 1 ≤ p < 2 with O(1)-distortion and dimension independent of n.

Empirical Performance of Approximate Algorithms for Low Rank Approximation

- Computer Science
- 2019

This data analysis project studies the empirical performance of state-of-the-art in theory approximation algorithms for the `F and `1 variants of the low-rank approximation problem on a variety of synthetic and real datasets.

Optimal CUR matrix decompositions

- Computer Science, MathematicsSTOC
- 2014

This work presents input-sparsity-time and deterministic algorithms for constructing such a CUR matrix decomposition of A where c = O(k/ε) and r = O (k/ ε) and rank(U) = k and the construction is simultaneously optimal in c, r, andRank(U).

Weighted low rank approximations with provable guarantees

- Computer ScienceSTOC
- 2016

The present paper is the first to provide algorithms for the weighted low rank approximation problem with provable guarantees, and this technique turns out to be general enough to give solutions to several other fundamental problems: adversarial matrix completion, weighted non-negative matrix factorization and tensor completion.

## References

SHOWING 1-10 OF 46 REFERENCES

Low-distortion subspace embeddings in input-sparsity time and applications to robust linear regression

- Computer ScienceSTOC '13
- 2013

These input-sparsity time l<sub>p</sub> embeddings are optimal, up to constants, in terms of their running time; and the improved running time propagates to applications such as (1 pm ε)-distortion l< sub>p </sub> subspace embedding and relative-error l< Sub>p.</sub> regression.

OSNAP: Faster Numerical Linear Algebra Algorithms via Sparser Subspace Embeddings

- Computer Science2013 IEEE 54th Annual Symposium on Foundations of Computer Science
- 2013

The main result is essentially a Bai-Yin type theorem in random matrix theory and is likely to be of independent interest: for any fixed U ∈ R<sup>n×d</sup> with orthonormal columns and random sparse Π, all singular values of ΠU lie in [1 - ε, 1 + ε] with good probability.

Improved Approximation Algorithms for Large Matrices via Random Projections

- Computer Science2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS'06)
- 2006

The key idea is that low dimensional embeddings can be used to eliminate data dependence and provide more versatile, linear time pass efficient matrix computation.

Fast monte-carlo algorithms for finding low-rank approximations

- Computer ScienceJACM
- 2004

An algorithm is developed that is qualitatively faster, provided the authors may sample the entries of the matrix in accordance with a natural probability distribution, and implies that in constant time, it can be determined if a given matrix of arbitrary size has a good low-rank approximation.

Adaptive Sampling and Fast Low-Rank Matrix Approximation

- Computer Science, MathematicsAPPROX-RANDOM
- 2006

We prove that any real matrix A contains a subset of at most 4k/e+ 2k log(k+1) rows whose span “contains” a matrix of rank at most k with error only (1+e) times the error of the best rank-k…

Matrix approximation and projective clustering via volume sampling

- Computer Science, MathematicsSODA 2006
- 2006

This paper proves that the additive error drops exponentially by iterating the sampling in an adaptive manner, and gives a pass-efficient algorithm for computing low-rank approximation with reduced additive error.

Sampling from large matrices: An approach through geometric functional analysis

- Computer ScienceJACM
- 2007

The result for the cut-norm yields a slight improvement on the best-known sample complexity for an approximation algorithm for MAX-2CSP problems, and the law of large numbers for operator-valued random variables for Banach spaces is used.

Sparser Johnson-Lindenstrauss Transforms

- Computer ScienceJACM
- 2014

These are the first constructions to provide subconstant sparsity for all values of parameters, improving upon previous works of Achlioptas and Dasgupta et al.

Fast computation of low-rank matrix approximations

- Computer ScienceJACM
- 2007

It is proved that the effect of sampling and quantization nearly vanishes when a low-rank approximation to A + N is computed, which gives high probability bounds on the quality of the approximation both in the Frobenius and the 2-norm.

Subspace Sampling and Relative-Error Matrix Approximation: Column-Based Methods

- Computer Science, MathematicsAPPROX-RANDOM
- 2006

Two polynomial time randomized algorithms that take as input a matrix A and return as output a matrix C sample the columns of A via the method of “subspace sampling,” so-named since the sampling probabilities depend on the lengths of the rows of the top singular vectors and since they ensure that they capture entirely a certain subspace of interest.