• Corpus ID: 249494457

Deeper-GXX: Deepening Arbitrary GNNs

@inproceedings{Zheng2021DeeperGXXDA,
  title={Deeper-GXX: Deepening Arbitrary GNNs},
  author={Lecheng Zheng and Dongqi Fu and Ross Maciejewski and Jingrui He},
  year={2021}
}
Graph neural networks (GNNs) have proven successful at modeling graph data. However, shallow GNNs tend to have sub-optimal performance, e.g., dealing with large graphs with missing features. Therefore, it is necessary to increase the number of layers of GNNs to capture more latent knowledge of the input data. Nevertheless, stacking more layers in GNNs typically decreases their performance due to, e.g., vanishing gradient and oversmoothing. Existing deep GNN solutions mainly focus on addressing… 
[PDF] Semantic Reader
4 Citations
Background Citations
1

Figures and Tables from this paper

4 Citations

Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study

This work presents the first fair and reproducible benchmark dedicated to assessing the “tricks” of training deep GNNs, and demonstrates that an organic combo of initial connection, identity mapping, group and batch normalization attains the new state-of-the-art results for deep Gnns on large datasets.

MentorGNN: Deriving Curriculum for Pre-Training GNNs

An end-to-end model named MentorGNN that aims to supervise the pre-training process of GNNs across graphs with diverse structures and disparate feature spaces is proposed and new light is shed on the problem of domain adaption on relational data by deriving a natural and interpretable upper bound on the generalization error of thePre-trained Gnns.

X-GOAL: Multiplex Heterogeneous Graph Prototypical Contrastive Learning

A novel multiplex heterogeneous graph prototypical contrastive leaning (X-GOAL) framework to extract node embeddings is proposed, comprised of two components: the GOAL framework, which learns nodeembeddings for each homogeneous graph layer, and an alignment regularization, which jointly models different layers by aligning layer-specific node embeds.

Graph Augmentation Learning

In this survey, an in-depth review GAL techniques from macro (graph), meso (subgraph), and micro (node/edge) levels are reviewed and insights on several open issues of GAL are shared, including heterogeneity, spatio-temporal dynamics, scalability, and generalization.

Contrastive Learning with Complex Heterogeneity

A unified heterogeneous learning framework is proposed, which combines both the weighted unsupervised contrastive loss and the weighted supervised Contrastive loss to model multiple types of heterogeneity.

Natural and Artificial Dynamics in Graphs: Concept, Progress, and Future

The definitions of natural dynamics and artificial dynamics in graphs are introduced, and the related works of natural andificial dynamics about how they boost the aforementioned graph research topics are discussed.

Natural and Artificial Dynamics in GNNs: A Tutorial

This tutorial focuses on topics of natural dynamics and artificial dynamics in GNNs and introduces the related works systematically and point out some promising but under-explored research problems in the combination of these two dynamics.

Fairness-aware Multi-view Clustering

A fairness-aware multi-view clustering method that incorporates the group fairness constraint into the soft membership assignment for each cluster to ensure that the fraction of different groups in each cluster is approximately identical to the entire data set is proposed.

DISCO: Comprehensive and Explainable Disinformation Detection

A comprehensive and explainable disinformation detection framework called DISCO, which leverages the heterogeneity of disinformation and addresses the opaqueness of prediction, is proposed.

References

SHOWING 1-10 OF 44 REFERENCES

PairNorm: Tackling Oversmoothing in GNNs

PairNorm is a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar and significantly boosts performance for a new problem setting that benefits from deeper GNNs.

DropEdge: Towards Deep Graph Convolutional Networks on Node Classification

DropEdge is a general skill that can be equipped with many other backbone models (e.g. GCN, ResGCN, GraphSAGE, and JKNet) for enhanced performance and consistently improves the performance on a variety of both shallow and deep GCNs.

Graph Contrastive Learning with Augmentations

The results show that, even without tuning augmentation extents nor using sophisticated GNN architectures, the GraphCL framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods.

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning

It is shown that the graph convolution of the GCN model is actually a special form of Laplacian smoothing, which is the key reason why GCNs work, but it also brings potential concerns of over-smoothing with many convolutional layers.

Simple and Deep Graph Convolutional Networks

The GCNII is proposed, an extension of the vanilla GCN model with two simple yet effective techniques: {\em Initial residual} and {\em Identity mapping} that effectively relieves the problem of over-smoothing.

Simplifying Graph Convolutional Networks

This paper successively removes nonlinearities and collapsing weight matrices between consecutive layers, and theoretically analyze the resulting linear model and show that it corresponds to a fixed low-pass filter followed by a linear classifier.

Towards Deeper Graph Neural Networks with Differentiable Group Normalization

DGN is introduced, which normalizes nodes within the same group independently to increase their smoothness, and separates node distributions among different groups to significantly alleviate the over-smoothing issue.

Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View

Two methods to alleviate the over-smoothing issue of GNNs are proposed: MADReg which adds a MADGap-based regularizer to the training objective; AdaEdge which optimizes the graph topology based on the model predictions.

Graph Neural Networks Exponentially Lose Expressive Power for Node Classification

The theory enables us to relate the expressive power of GCNs with the topological information of the underlying graphs inherent in the graph spectra and provides a principled guideline for weight normalization of graph NNs.

Graph Attention Networks

We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior…