Efficient scaling of dynamic graph neural networks

@article{Chakaravarthy2021EfficientSO,
  title={Efficient scaling of dynamic graph neural networks},
  author={Venkatesan T. Chakaravarthy and Shivmaran S. Pandian and Saurabh Raje and Yogish Sabharwal and Toyotaro Suzumura and Shashanka Ubaru},
  journal={Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
  year={2021}
}
We present distributed algorithms for training dynamic Graph Neural Networks (GNN) on large scale graphs spanning multi-node, multi-GPU systems. To the best of our knowledge, this is the first scaling study on dynamic GNN. We devise mechanisms for reducing the GPU memory usage and identify two execution time bottlenecks: CPU-GPU data transfer; and communication volume. Exploiting properties of dynamic graphs, we design a graph difference-based strategy to significantly reduce the transfer time… Expand

Figures and Tables from this paper

References

SHOWING 1-10 OF 58 REFERENCES
NeuGraph: Parallel Deep Neural Network Computation on Large Graphs
TLDR
The evaluation shows that, on small graphs that can fit in a single GPU, NeuGraph outperforms state-of-the-art implementations by a significant margin, while scaling to large real-world graphs that none of the existing frameworks can handle directly with GPUs. Expand
Computing Graph Neural Networks: A Survey from Algorithms to Accelerators
TLDR
A review of the field of GNNs is presented from the perspective of computing, and an in-depth analysis of current software and hardware acceleration schemes is provided, from which a hardware-software, graph-aware, and communication-centric vision for GNN accelerators is distilled. Expand
Training Deep Nets with Sublinear Memory Cost
TLDR
This work designs an algorithm that costs O( √ n) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch, and shows that it is possible to trade computation for memory giving a more memory efficient training algorithm with a little extra computation cost. Expand
AGL: A Scalable System for Industrial-purpose Graph Machine Learning
TLDR
AGL is designed, a scalable, fault-tolerance and integrated system, with fully-functional training and inference for GNNs, implemented on mature infrastructures such as MapReduce. Expand
A Comprehensive Survey on Graph Neural Networks
TLDR
This article provides a comprehensive overview of graph neural networks (GNNs) in data mining and machine learning fields and proposes a new taxonomy to divide the state-of-the-art GNNs into four categories, namely, recurrent GNNS, convolutional GNN’s, graph autoencoders, and spatial–temporal Gnns. Expand
Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs
TLDR
Deep Graph Library (DGL) enables arbitrary message handling and mutation operators, flexible propagation rules, and is framework agnostic so as to leverage high-performance tensor, autograd operations, and other feature extraction modules already available in existing frameworks. Expand
Streaming Graph Neural Networks
TLDR
DyGNN, a Dynamic Graph Neural Network model, which can model the dynamic information as the graph evolving is proposed, which keeps updating node information by capturing the sequential information of edges (interactions), the time intervals between edges and information propagation coherently. Expand
Graph Convolutional Neural Networks for Web-Scale Recommender Systems
TLDR
A novel method based on highly efficient random walks to structure the convolutions and a novel training strategy that relies on harder-and-harder training examples to improve robustness and convergence of the model are developed. Expand
AliGraph: A Comprehensive Graph Neural Network Platform
TLDR
This paper presents a comprehensive graph neural network system, namely AliGraph, which consists of distributed graph storage, optimized sampling operators and runtime to efficiently support not only existing popular GNNs but also a series of in-house developed ones for different scenarios. Expand
EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs
TLDR
This work proposes EvolveGCN, which adapts the graph convolutional network (GCN) model along the temporal dimension without resorting to node embeddings, and captures the dynamism of the graph sequence through using an RNN to evolve the GCN parameters. Expand
...
1
2
3
4
5
...