Simple GNN Regularisation for 3D Molecular Property Prediction and Beyond
@inproceedings{Godwin2021SimpleGR, title={Simple GNN Regularisation for 3D Molecular Property Prediction and Beyond}, author={Jonathan Godwin and Michael Schaarschmidt and Alex Gaunt and Alvaro Sanchez-Gonzalez and Yulia Rubanova and Petar Velivckovi'c and James Kirkpatrick and Peter W. Battaglia}, booktitle={International Conference on Learning Representations}, year={2021} }
In this paper we show that simple noisy regularisation can be an effective way to address GNN oversmoothing. First we argue that regularisers addressing oversmoothing should both penalise node latent similarity and encourage meaningful node representations. From this observation we derive “Noisy Nodes”, a simple technique in which we corrupt the input graph with noise, and add a noise correcting node-level loss. The diverse node level loss encourages latent node diversity, and the denoising…
Figures and Tables from this paper
40 Citations
SupSiam: Non-contrastive Auxiliary Loss for Learning from Molecular Conformers
- Computer ScienceArXiv
- 2023
It is found that a non-contrastive (positive-pair only) auxiliary task aids in supervised training of Euclidean neural networks (E3NNs) and increases manifold smoothness (MS) around point-cloud geometries.
Pre-training via Denoising for Molecular Property Prediction
- Computer ScienceArXiv
- 2022
A pre-training technique based on denoising that achieves a new state-of-the-art in molecular property prediction by utilizing large datasets of 3D molecular structures at equilibrium to learn meaningful representations for downstream tasks.
Curvature-informed multi-task learning for graph networks
- Computer ScienceArXiv
- 2022
This work investigates a potential expla-nation for this phenomenon – the curvature of each property’s loss surface significantly varies, leading tocient learning.
Robust Optimization as Data Augmentation for Large-scale Graphs
- Computer Science2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- 2022
This work proposes FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training, which universally works in node classification, link prediction, and graph classification tasks.
Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching
- Computer ScienceArXiv
- 2022
GeoSSL is proposed, a 3D coordinate denoising pretraining framework to model such an energy landscape based on the dynamic nature of 3D molecules, where the continuous motion of a molecule in the 3D Euclidean space forms a smooth potential energy surface.
Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs
- Computer ScienceArXiv
- 2022
It is demonstrated that Transformers can generalize well to 3D atomistic graphs and Equiformer, a graph neural network leveraging the strength of Transformer architectures and incorporating SE(3)/E(3)-equivariant features based on irreducible representations (irreps), is presented.
ComENet: Towards Complete and Efficient Message Passing for 3D Molecular Graphs
- Computer ScienceArXiv
- 2022
This work proposes a novel message passing scheme that operates within 1-hop neighborhood that guarantees full completeness of 3D information on 3D graphs by achieving global and local completeness, and provides rigorous proof of completeness and analysis of time complexity for the methods.
Spherical Message Passing for 3D Molecular Graphs
- Computer ScienceICLR
- 2022
This work proposes the spherical message passing (SMP) as a novel and powerful scheme for 3D molecular learning that dramatically reduces training complexity, enabling it to perform efficiently on large-scale molecules and demonstrates the advantages of SphereNet in terms of capability, efficiency, and scalability.
Towards Training Billion Parameter Graph Neural Networks for Atomic Simulations
- Computer ScienceICLR
- 2022
Graph Parallelism is introduced, a method to distribute input graphs across multiple GPUs, enabling us to train very large GNNs with hundreds of millions or billions of parameters.
Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets
- Computer ScienceArXiv
- 2022
The recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecules are described, which show that with a global receptive and an adaptive aggregation strategy, graphormer is more powerful than classic message-passing-based GNNs.
References
SHOWING 1-10 OF 73 REFERENCES
PairNorm: Tackling Oversmoothing in GNNs
- Computer ScienceICLR
- 2020
PairNorm is a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar and significantly boosts performance for a new problem setting that benefits from deeper GNNs.
Revisiting "Over-smoothing" in Deep GCNs
- Computer ScienceArXiv
- 2020
This work interprets a standard GCN architecture as layerwise integration of a Multi-layer Perceptron (MLP) and graph regularization and concludes that before training, the final representation of a deep GCN does over-smooth, however, it learns anti-oversmoothing during training.
Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View
- Computer ScienceAAAI
- 2020
Two methods to alleviate the over-smoothing issue of GNNs are proposed: MADReg which adds a MADGap-based regularizer to the training objective; AdaEdge which optimizes the graph topology based on the model predictions.
Relational Pooling for Graph Representations
- Computer ScienceICML
- 2019
This work generalizes graph neural networks (GNNs) beyond those based on the Weisfeiler-Lehman (WL) algorithm, graph Laplacians, and diffusions to provide a framework with maximal representation power for graphs.
The Truly Deep Graph Convolutional Networks for Node Classification
- Computer ScienceArXiv
- 2019
DropEdge is proposed, a novel technique that randomly removes a certain number of edges from the input graphs, acting like a data augmenter and also a message passing reducer, and enables us to recast a wider range of Convolutional Neural Networks from the image field to the graph domain.
A Note on Over-Smoothing for Graph Neural Networks
- Computer ScienceArXiv
- 2020
It is shown that when the weight matrix satisfies the conditions determined by the spectrum of augmented normalized Laplacian, the Dirichlet energy of embeddings will converge to zero, resulting in the loss of discriminative power.
Neural Message Passing for Quantum Chemistry
- Computer ScienceICML
- 2017
Using MPNNs, state of the art results on an important molecular property prediction benchmark are demonstrated and it is believed future work should focus on datasets with larger molecules or more accurate ground truth labels.
Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
- Computer ScienceAAAI
- 2018
It is shown that the graph convolution of the GCN model is actually a special form of Laplacian smoothing, which is the key reason why GCNs work, but it also brings potential concerns of over-smoothing with many convolutional layers.
FLAG: Adversarial Data Augmentation for Graph Neural Networks
- Computer ScienceArXiv
- 2020
This work proposes a simple but effective solution, FLAG (Free Large-scale Adversarial Augmentation on Graphs), which iteratively augments node features with gradient-based adversarial perturbations during training, and boosts performance at test time.
Graph Contrastive Learning with Augmentations
- Computer ScienceNeurIPS
- 2020
The results show that, even without tuning augmentation extents nor using sophisticated GNN architectures, the GraphCL framework can produce graph representations of similar or better generalizability, transferrability, and robustness compared to state-of-the-art methods.