# Training Graph Neural Networks with 1000 Layers

@inproceedings{Li2021TrainingGN, title={Training Graph Neural Networks with 1000 Layers}, author={Guohao Li and Matthias M{\"u}ller and Bernard Ghanem and V. Koltun}, booktitle={ICML}, year={2021} }

Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges. However, memory complexity has become a major obstacle when training deep GNNs for practical applications due to the immense number of nodes, edges, and intermediate activations. To improve the scalability of GNNs, prior works propose smart graph sampling or partitioning strategies to train GNNs with a smaller set of nodes or sub-graphs. In… Expand

#### Figures and Tables from this paper

#### 2 Citations

Evaluating Deep Graph Neural Networks

- Computer Science
- ArXiv
- 2021

The first systematic experimental evaluation is conducted to present the fundamental limitations of shallow architectures and presents Deep Graph Multi-Layer Perceptron (DGMLP), a powerful approach (a paradigm in its own right) that helps guide deep GNN designs. Expand

Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networks

- Computer Science
- 2021

ZHIQIAN CHEN, Dept. of Computer Science and Engineering, Mississippi State University, U.S.A FANGLAN CHEN, Dept. of Computer Science, Virginia Tech, U.S.A LEI ZHANG, Dept. of Computer Science,… Expand

#### References

SHOWING 1-10 OF 78 REFERENCES

Scaling Graph Neural Networks with Approximate PageRank

- Computer Science, Mathematics
- KDD
- 2020

The PPRGo model is presented, which utilizes an efficient approximation of information diffusion in GNNs resulting in significant speed gains while maintaining state-of-the-art prediction performance, and the practical application of PPR go to solve large-scale node classification problems at Google. Expand

How Powerful are Graph Neural Networks?

- Computer Science, Mathematics
- ICLR
- 2019

This work characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures, and develops a simple architecture that is provably the most expressive among the class of GNNs. Expand

Pitfalls of Graph Neural Network Evaluation

- Computer Science, Mathematics
- ArXiv
- 2018

This paper performs a thorough empirical evaluation of four prominent GNN models and suggests that simpler GNN architectures are able to outperform the more sophisticated ones if the hyperparameters and the training procedure are tuned fairly for all models. Expand

DeeperGCN: All You Need to Train Deeper GCNs

- Computer Science, Mathematics
- ArXiv
- 2020

Extensive experiments on Open Graph Benchmark show DeeperGCN significantly boosts performance over the state-of-the-art on the large scale graph learning tasks of node property prediction and graph property prediction. Expand

Simple and Deep Graph Convolutional Networks

- Computer Science, Mathematics
- ICML
- 2020

The GCNII is proposed, an extension of the vanilla GCN model with two simple yet effective techniques: {\em Initial residual} and {\em Identity mapping} that effectively relieves the problem of over-smoothing. Expand

Large-Scale Learnable Graph Convolutional Networks

- Computer Science, Mathematics
- KDD
- 2018

The proposed LGCL automatically selects a fixed number of neighboring nodes for each feature based on value ranking in order to transform graph data into grid-like structures in 1-D format, thereby enabling the use of regular convolutional operations on generic graphs. Expand

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

- Computer Science
- ICLR
- 2020

DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance and consistently improves the performance on a variety of both shallow and deep GCNs. Expand

Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

- Computer Science, Mathematics
- KDD
- 2019

Cluster-GCN is proposed, a novel GCN algorithm that is suitable for SGD-based training by exploiting the graph clustering structure and allows us to train much deeper GCN without much time and memory overhead, which leads to improved prediction accuracy. Expand

Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth

- Computer Science, Mathematics
- ICML
- 2021

This work analyzes linearized GNNs and proves that despite the non-convexity of training, convergence to a global minimum at a linear rate is guaranteed under mild assumptions that are validated on real-world graphs. Expand

Graph Attention Networks

- Mathematics, Computer Science
- ICLR
- 2018

We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior… Expand