Corpus ID: 237940251

Communication-Efficient Distributed Linear and Deep Generalized Canonical Correlation Analysis

@article{Shrestha2021CommunicationEfficientDL,
  title={Communication-Efficient Distributed Linear and Deep Generalized Canonical Correlation Analysis},
  author={Sagar Shrestha and Xiao Fu},
  journal={ArXiv},
  year={2021},
  volume={abs/2109.12400}
}
  • Sagar Shrestha, Xiao Fu
  • Published 25 September 2021
  • Computer Science, Engineering
  • ArXiv
Classic and deep learning-based generalized canonical correlation analysis (GCCA) algorithms seek low-dimensional common representations of data entities from multiple “views” (e.g., audio and image) using linear transformations and neural networks, respectively. When the views are acquired and stored at different locations, organizations and edge devices, computing GCCA in a distributed, parallel and efficient manner is wellmotivated. However, existing distributed GCCA algorithms may incur… Expand

References

SHOWING 1-10 OF 40 REFERENCES
COMMUNICATION-EFFICIENT DISTRIBUTED MAX-VAR GENERALIZED CCA VIA ERROR FEEDBACK
  • Sagar Shrestha, Xiao Fu
  • 2021
Generalized Canonical Correlation Analysis (GCCA) learns maximally correlated low-dimensional representation from multiple “views” of the data (e.g., audio and video of the same event). GCCA entailsExpand
Efficient and Distributed Algorithms for Large-Scale Generalized Canonical Correlations Analysis
TLDR
This work proposes a GCCA algorithm whose memory and computational costs scale linearly in the problem dimension and the number of nonzero data elements, respectively, which can easily handle very large sparse views whose sample and feature dimensions both exceed 100,000 – while the current approaches can only handle thousands of features / samples. Expand
QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
TLDR
Quantized SGD is proposed, a family of compression schemes for gradient updates which provides convergence guarantees and leads to significant reductions in end-to-end training time, and can be extended to stochastic variance-reduced techniques. Expand
Scalable and Flexible Multiview MAX-VAR Canonical Correlation Analysis
TLDR
This work posit regularized MAX-VAR GCCA as a nonconvex optimization problem and proposes an alternating optimization-based algorithm to handle it, which alternates between inexact solutions of a regularized least squares subproblem and a manifold-constrained non Convex subproblem, thereby achieving substantial memory and computational savings. Expand
Structured SUMCOR Multiview Canonical Correlation Analysis for Large-Scale Data
TLDR
This work proposes a new computational framework for large-scale SUMCOR GCCA that can easily incorporate a suite of structural regularizers, which are frequently used in data analytics and shows that the proposed algorithm converges to a Karush–Kuhn–Tucker point of the regularized SUMCOR problem. Expand
Qsparse-Local-SGD: Distributed SGD With Quantization, Sparsification, and Local Computations
TLDR
This paper proposes Qsparse-local-SGD algorithm, which combines aggressive sparsification with quantization and local computation along with error compensation, by keeping track of the difference between the true and compressed gradients, and demonstrates that it converges at the same rate as vanilla distributed SGD for many important classes of sparsifiers and quantizers. Expand
Deep Canonical Correlation Analysis
TLDR
DCCA is introduced, a method to learn complex nonlinear transformations of two views of data such that the resulting representations are highly linearly correlated and Parameters of both transformations are jointly learned to maximize the (regularized) total correlation. Expand
Sparsified SGD with Memory
TLDR
This work analyzes Stochastic Gradient Descent with k-sparsification or compression (for instance top-k or random-k) and shows that this scheme converges at the same rate as vanilla SGD when equipped with error compensation. Expand
Distributed Canonical Correlation Analysis in Wireless Sensor Networks With Application to Distributed Blind Source Separation
TLDR
This paper proposes a distributed CCA (DCCA) algorithm that can operate in wireless sensor networks (WSNs) with a fully connected or a tree topology and proves convergence and optimality of the DCCA algorithm. Expand
signSGD: compressed optimisation for non-convex problems
TLDR
SignSGD can get the best of both worlds: compressed gradients and SGD-level convergence rate, and the momentum counterpart of signSGD is able to match the accuracy and convergence speed of Adam on deep Imagenet models. Expand
...
1
2
3
4
...