• Corpus ID: 20623756

Coding Method for Parallel Iterative Linear Solver

@article{Yang2017CodingMF,
  title={Coding Method for Parallel Iterative Linear Solver},
  author={Yaoqing Yang and Pulkit Grover and Soummya Kar},
  journal={ArXiv},
  year={2017},
  volume={abs/1706.00163}
}
Computationally intensive distributed and parallel computing is often bottlenecked by a small set of slow workers known as stragglers. In this paper, we utilize the emerging idea of "coded computation" to design a novel error-correcting-code inspired technique for solving linear inverse problems under specific iterative methods in a parallelized implementation affected by stragglers. Example applications include inverse problems in machine learning on graphs, such as personalized PageRank and… 

Figures from this paper

Distributed Matrix Multiplication Using Speed Adaptive Coding

A dynamic workload distribution strategy for coded computation called Slack Squeeze Coded Computation ($S^2C^2) that squeezes the compute slack (i.e., overhead) that is built into the coded computing frameworks by efficiently assigning work for all fast and slow nodes according to their speeds and without needing to re-distribute data.

Straggler-Proofing Massive-Scale Distributed Matrix Multiplication with D-Dimensional Product Codes

This work presents a novel coded matrix-matrix multiplication scheme based on d-dimensional product codes that allows for order-optimal computation/communication costs for the encoding/decoding procedures while achieving near-Optimal compute time.

Matrix sparsification for coded matrix multiplication

This work shows that the Short-Dot scheme is optimal if an Maximum Distance Separable (MDS) matrix is fixed, and proposes a new encoding scheme that can achieve a strictly larger sparsity than the existing schemes.

Slack squeeze coded computing for adaptive straggler mitigation

This paper proposes a dynamic workload distribution strategy for coded computation called Slack Squeeze Coded Computation (S2C2), which squeezes the compute slack (i.e., overhead) that is built into the coded computing frameworks by efficiently assigning work for all fast and slow nodes according to their speeds and without needing to re-distribute data.

Latency analysis of coded computation schemes over wireless networks

This paper considers three asymptotic regimes (determined by how the communication and computation times are scaled with the number of workers) and precisely characterize the expected total run-time of the distributed algorithm and optimum coding strategy in each regime and demonstrates that coded schemes are Θ(logn) times faster than uncoded schemes.

Communication-Computation Efficient Gradient Coding

This paper develops coding techniques to reduce the running time of distributed learning tasks by giving an explicit coding scheme that achieves the optimal tradeoff based on recursive polynomial constructions, coding both across data subsets and vector components.

Robust Gradient Descent via Moment Encoding with LDPC Codes

It is shown that for a random model for stragglers, the proposed moment encoding based gradient descent method can be viewed as the stochastic gradient descent Method, which allows for convergence guarantees for the proposed solution.

Robust Gradient Descent via Moment Encoding and LDPC Codes

This paper proposes to encode the second-moment of the data with a low density parity-check (LDPC) code, a iterative decoding algorithm for LDPC codes that has very low computational overhead, and outperforms the existing schemes in a real distributed computing setup.

On the optimal recovery threshold of coded matrix multiplication

We provide novel coded computation strategies for distributed matrix-matrix products that outperform the recent “Polynomial code” constructions in recovery threshold, i.e., the required number of

Train Where the Data is: A Case for Bandwidth Efficient Coded Training

A Random Linear Network coding (RLNC) is used which reduces the need to exchange data partitions across all participating mobile devices, while at the same time preserving the property of coded computing to tolerate uncertainties.

References

SHOWING 1-10 OF 44 REFERENCES

Anytime coding for distributed computation

  • Nuwan S. FerdinandS. Draper
  • Computer Science
    2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
  • 2016
A novel coding scheme is proposed to speed up distributed computation through a form of approximate computing, in a general framework in terms of the expected cost of the approximate solution and in the context of vector-matrix multiplication.

Coded computation for multicore setups

This work proposes the use of sparse linear codes, exploiting the modern multicore processing architecture, and shows that its coding solution achieves the order optimal runtime, and is at least Θ(√log n) times faster than any uncoded schemes where the number of workers is n.

Polynomial Codes: an Optimal Design for High-Dimensional Coded Matrix Multiplication

We consider a large-scale matrix multiplication problem where the computation is carried out using a distributed system with a master node and multiple worker nodes, where each worker can store parts

Coded computation over heterogeneous clusters

This paper proposes Heterogeneous Coded Matrix Multiplication (HCMM) algorithm for performing distributed matrix multiplication over heterogeneous clusters that is provably asymptotically optimal and provides numerical results demonstrating significant speedups of up to 49% and 34% for HCMM in comparison to the “uncoded” and “homogeneous coded” schemes.

High-dimensional coded matrix multiplication

This work studies coded computation involving large matrix multiplication where both matrices are large, and proposes a new coded computation scheme, which is called product-coded matrix multiplication, which reveals interesting insights into which schemes perform best in which regimes.

On the worst-case communication overhead for distributed data shuffling

  • M. AttiaR. Tandon
  • Computer Science
    2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
  • 2016
This paper proposes a novel coded data delivery scheme for the case of no excess storage, where every worker can only store the assigned data batches under processing, and is applicable to any arbitrary shuffle, and for any number of workers.

Speeding Up Distributed Machine Learning Using Codes

This paper focuses on two of the most basic building blocks of distributed learning algorithms: matrix multiplication and data shuffling, and uses codes to reduce communication bottlenecks, exploiting the excess in storage.

Algorithm-Based Fault Tolerance for Matrix Operations

Algorithm-based fault tolerance schemes are proposed to detect and correct errors when matrix operations such as addition, multiplication, scalar product, LU-decomposition, and transposition are performed using multiple processor systems.

Using Straggler Replication to Reduce Latency in Large-scale Parallel Computing

This work analyses how task replication reduces latency, and proposes a heuristic algorithm to search for the best replication strategies when it is difficult to model the empirical behavior of task execution time and uses the proposed analysis techniques.

Decentralized coded caching attains order-optimal memory-rate tradeoff

  • M. Maddah-AliUrs Niesen
  • Computer Science
    2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton)
  • 2013
This paper proposes an efficient caching scheme, in which the content placement is performed in a decentralized manner, and hence achieves a rate close to the centralized scheme.