# Coding Method for Parallel Iterative Linear Solver

@article{Yang2017CodingMF, title={Coding Method for Parallel Iterative Linear Solver}, author={Yaoqing Yang and Pulkit Grover and Soummya Kar}, journal={ArXiv}, year={2017}, volume={abs/1706.00163} }

Computationally intensive distributed and parallel computing is often bottlenecked by a small set of slow workers known as stragglers. In this paper, we utilize the emerging idea of "coded computation" to design a novel error-correcting-code inspired technique for solving linear inverse problems under specific iterative methods in a parallelized implementation affected by stragglers. Example applications include inverse problems in machine learning on graphs, such as personalized PageRank and…

## 11 Citations

### Distributed Matrix Multiplication Using Speed Adaptive Coding

- Computer ScienceArXiv
- 2019

A dynamic workload distribution strategy for coded computation called Slack Squeeze Coded Computation ($S^2C^2) that squeezes the compute slack (i.e., overhead) that is built into the coded computing frameworks by efficiently assigning work for all fast and slow nodes according to their speeds and without needing to re-distribute data.

### Straggler-Proofing Massive-Scale Distributed Matrix Multiplication with D-Dimensional Product Codes

- Computer Science2018 IEEE International Symposium on Information Theory (ISIT)
- 2018

This work presents a novel coded matrix-matrix multiplication scheme based on d-dimensional product codes that allows for order-optimal computation/communication costs for the encoding/decoding procedures while achieving near-Optimal compute time.

### Matrix sparsification for coded matrix multiplication

- Computer Science2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
- 2017

This work shows that the Short-Dot scheme is optimal if an Maximum Distance Separable (MDS) matrix is fixed, and proposes a new encoding scheme that can achieve a strictly larger sparsity than the existing schemes.

### Slack squeeze coded computing for adaptive straggler mitigation

- Computer ScienceSC
- 2019

This paper proposes a dynamic workload distribution strategy for coded computation called Slack Squeeze Coded Computation (S2C2), which squeezes the compute slack (i.e., overhead) that is built into the coded computing frameworks by efficiently assigning work for all fast and slow nodes according to their speeds and without needing to re-distribute data.

### Latency analysis of coded computation schemes over wireless networks

- Computer Science2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
- 2017

This paper considers three asymptotic regimes (determined by how the communication and computation times are scaled with the number of workers) and precisely characterize the expected total run-time of the distributed algorithm and optimum coding strategy in each regime and demonstrates that coded schemes are Θ(logn) times faster than uncoded schemes.

### Communication-Computation Efficient Gradient Coding

- Computer ScienceICML
- 2018

This paper develops coding techniques to reduce the running time of distributed learning tasks by giving an explicit coding scheme that achieves the optimal tradeoff based on recursive polynomial constructions, coding both across data subsets and vector components.

### Robust Gradient Descent via Moment Encoding with LDPC Codes

- Computer ScienceArXiv
- 2018

It is shown that for a random model for stragglers, the proposed moment encoding based gradient descent method can be viewed as the stochastic gradient descent Method, which allows for convergence guarantees for the proposed solution.

### Robust Gradient Descent via Moment Encoding and LDPC Codes

- Computer Science2019 IEEE International Symposium on Information Theory (ISIT)
- 2019

This paper proposes to encode the second-moment of the data with a low density parity-check (LDPC) code, a iterative decoding algorithm for LDPC codes that has very low computational overhead, and outperforms the existing schemes in a real distributed computing setup.

### On the optimal recovery threshold of coded matrix multiplication

- Computer Science2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
- 2017

We provide novel coded computation strategies for distributed matrix-matrix products that outperform the recent “Polynomial code” constructions in recovery threshold, i.e., the required number of…

### Train Where the Data is: A Case for Bandwidth Efficient Coded Training

- Computer ScienceArXiv
- 2019

A Random Linear Network coding (RLNC) is used which reduces the need to exchange data partitions across all participating mobile devices, while at the same time preserving the property of coded computing to tolerate uncertainties.

## References

SHOWING 1-10 OF 44 REFERENCES

### Anytime coding for distributed computation

- Computer Science2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
- 2016

A novel coding scheme is proposed to speed up distributed computation through a form of approximate computing, in a general framework in terms of the expected cost of the approximate solution and in the context of vector-matrix multiplication.

### Coded computation for multicore setups

- Computer Science2017 IEEE International Symposium on Information Theory (ISIT)
- 2017

This work proposes the use of sparse linear codes, exploiting the modern multicore processing architecture, and shows that its coding solution achieves the order optimal runtime, and is at least Θ(√log n) times faster than any uncoded schemes where the number of workers is n.

### Polynomial Codes: an Optimal Design for High-Dimensional Coded Matrix Multiplication

- Computer ScienceNIPS
- 2017

We consider a large-scale matrix multiplication problem where the computation is carried out using a distributed system with a master node and multiple worker nodes, where each worker can store parts…

### Coded computation over heterogeneous clusters

- Computer Science2017 IEEE International Symposium on Information Theory (ISIT)
- 2017

This paper proposes Heterogeneous Coded Matrix Multiplication (HCMM) algorithm for performing distributed matrix multiplication over heterogeneous clusters that is provably asymptotically optimal and provides numerical results demonstrating significant speedups of up to 49% and 34% for HCMM in comparison to the “uncoded” and “homogeneous coded” schemes.

### High-dimensional coded matrix multiplication

- Computer Science2017 IEEE International Symposium on Information Theory (ISIT)
- 2017

This work studies coded computation involving large matrix multiplication where both matrices are large, and proposes a new coded computation scheme, which is called product-coded matrix multiplication, which reveals interesting insights into which schemes perform best in which regimes.

### On the worst-case communication overhead for distributed data shuffling

- Computer Science2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
- 2016

This paper proposes a novel coded data delivery scheme for the case of no excess storage, where every worker can only store the assigned data batches under processing, and is applicable to any arbitrary shuffle, and for any number of workers.

### Speeding Up Distributed Machine Learning Using Codes

- Computer ScienceIEEE Transactions on Information Theory
- 2018

This paper focuses on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling, and uses codes to reduce communication bottlenecks, exploiting the excess in storage.

### Algorithm-Based Fault Tolerance for Matrix Operations

- Computer ScienceIEEE Transactions on Computers
- 1984

Algorithm-based fault tolerance schemes are proposed to detect and correct errors when matrix operations such as addition, multiplication, scalar product, LU-decomposition, and transposition are performed using multiple processor systems.

### Using Straggler Replication to Reduce Latency in Large-scale Parallel Computing

- Computer SciencePERV
- 2015

This work analyses how task replication reduces latency, and proposes a heuristic algorithm to search for the best replication strategies when it is difficult to model the empirical behavior of task execution time and uses the proposed analysis techniques.

### Decentralized coded caching attains order-optimal memory-rate tradeoff

- Computer Science2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton)
- 2013

This paper proposes an efficient caching scheme, in which the content placement is performed in a decentralized manner, and hence achieves a rate close to the centralized scheme.