• Corpus ID: 247450503

Simple GNN Regularisation for 3D Molecular Property Prediction and Beyond

@inproceedings{Godwin2021SimpleGR,
  title={Simple GNN Regularisation for 3D Molecular Property Prediction and Beyond},
  author={Jonathan Godwin and Michael Schaarschmidt and Alex Gaunt and Alvaro Sanchez-Gonzalez and Yulia Rubanova and Petar Velivckovi'c and James Kirkpatrick and Peter W. Battaglia},
  booktitle={International Conference on Learning Representations},
  year={2021}
}
In this paper we show that simple noisy regularisation can be an effective way to address GNN oversmoothing. First we argue that regularisers addressing oversmoothing should both penalise node latent similarity and encourage meaningful node representations. From this observation we derive “Noisy Nodes”, a simple technique in which we corrupt the input graph with noise, and add a noise correcting node-level loss. The diverse node level loss encourages latent node diversity, and the denoising… 
[PDF] Semantic Reader
40 Citations
Highly Influential Citations
8
Background Citations
22
Methods Citations
18
Results Citations
2

40 Citations

SupSiam: Non-contrastive Auxiliary Loss for Learning from Molecular Conformers

It is found that a non-contrastive (positive-pair only) auxiliary task aids in supervised training of Euclidean neural networks (E3NNs) and increases manifold smoothness (MS) around point-cloud geometries.

Pre-training via Denoising for Molecular Property Prediction

A pre-training technique based on denoising that achieves a new state-of-the-art in molecular property prediction by utilizing large datasets of 3D molecular structures at equilibrium to learn meaningful representations for downstream tasks.

Curvature-informed multi-task learning for graph networks

This work investigates a potential expla-nation for this phenomenon – the curvature of each property’s loss surface significantly varies, leading tocient learning.

Robust Optimization as Data Augmentation for Large-scale Graphs

  • Kezhi KongG. Li T. Goldstein
  • Computer Science
    2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • 2022
This work proposes FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training, which universally works in node classification, link prediction, and graph classification tasks.

Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching

GeoSSL is proposed, a 3D coordinate denoising pretraining framework to model such an energy landscape based on the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface.

Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs

It is demonstrated that Transformers can generalize well to 3D atomistic graphs and Equiformer, a graph neural network leveraging the strength of Transformer architectures and incorporating SE(3)/E(3)-equivariant features based on irreducible representations (irreps), is presented.

ComENet: Towards Complete and Efficient Message Passing for 3D Molecular Graphs

This work proposes a novel message passing scheme that operates within 1-hop neighborhood that guarantees full completeness of 3D information on 3D graphs by achieving global and local completeness, and provides rigorous proof of completeness and analysis of time complexity for the methods.

Spherical Message Passing for 3D Molecular Graphs

This work proposes the spherical message passing (SMP) as a novel and powerful scheme for 3D molecular learning that dramatically reduces training complexity, enabling it to perform efficiently on large-scale molecules and demonstrates the advantages of SphereNet in terms of capability, efficiency, and scalability.

Towards Training Billion Parameter Graph Neural Networks for Atomic Simulations

Graph Parallelism is introduced, a method to distribute input graphs across multiple GPUs, enabling us to train very large GNNs with hundreds of millions or billions of parameters.

Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets

The recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecules are described, which show that with a global receptive and an adaptive aggregation strategy, graphormer is more powerful than classic message-passing-based GNNs.

References

SHOWING 1-10 OF 73 REFERENCES

PairNorm: Tackling Oversmoothing in GNNs

PairNorm is a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar and significantly boosts performance for a new problem setting that benefits from deeper GNNs.

Revisiting "Over-smoothing" in Deep GCNs

This work interprets a standard GCN architecture as layerwise integration of a Multi-layer Perceptron (MLP) and graph regularization and concludes that before training, the final representation of a deep GCN does over-smooth, however, it learns anti-oversmoothing during training.

Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View

Two methods to alleviate the over-smoothing issue of GNNs are proposed: MADReg which adds a MADGap-based regularizer to the training objective; AdaEdge which optimizes the graph topology based on the model predictions.

Relational Pooling for Graph Representations

This work generalizes graph neural networks (GNNs) beyond those based on the Weisfeiler-Lehman (WL) algorithm, graph Laplacians, and diffusions to provide a framework with maximal representation power for graphs.

The Truly Deep Graph Convolutional Networks for Node Classification

DropEdge is proposed, a novel technique that randomly removes a certain number of edges from the input graphs, acting like a data augmenter and also a message passing reducer, and enables us to recast a wider range of Convolutional Neural Networks from the image field to the graph domain.

A Note on Over-Smoothing for Graph Neural Networks

It is shown that when the weight matrix satisfies the conditions determined by the spectrum of augmented normalized Laplacian, the Dirichlet energy of embeddings will converge to zero, resulting in the loss of discriminative power.

Neural Message Passing for Quantum Chemistry

Using MPNNs, state of the art results on an important molecular property prediction benchmark are demonstrated and it is believed future work should focus on datasets with larger molecules or more accurate ground truth labels.

Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning

It is shown that the graph convolution of the GCN model is actually a special form of Laplacian smoothing, which is the key reason why GCNs work, but it also brings potential concerns of over-smoothing with many convolutional layers.

FLAG: Adversarial Data Augmentation for Graph Neural Networks

This work proposes a simple but effective solution, FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training, and boosts performance at test time.

Graph Contrastive Learning with Augmentations

The results show that, even without tuning augmentation extents nor using sophisticated GNN architectures, the GraphCL framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods.
...