# A Biased Graph Neural Network Sampler with Near-Optimal Regret

@inproceedings{Zhang2021ABG, title={A Biased Graph Neural Network Sampler with Near-Optimal Regret}, author={Qingru Zhang and David Paul Wipf and Quan Gan and Le Song}, booktitle={Neural Information Processing Systems}, year={2021} }

Graph neural networks (GNN) have recently emerged as a vehicle for applying deep network architectures to graph and relational data. However, given the increasing size of industrial datasets, in many practical situations the message passing computations required for sharing information across GNN layers are no longer scalable. Although various sampling methods have been introduced to approximate full-graph training within a tractable budget, there remain unresolved complications such as high…

## 10 Citations

### Graph Belief Propagation Networks

- Computer ScienceArXiv
- 2021

This work introduces a model that combines the advantages of these two approaches, where the marginal probabilities in a conditional random field, similar to collective classification, and the potentials in the random field are learned through end-to-end training, akin to graph neural networks.

### (LA)yer-neigh(BOR) Sampling: Defusing Neighborhood Explosion in GNNs

- Computer ScienceArXiv
- 2022

A new sampling algorithm called LAyer-neighBOR sampling (LABOR) is proposed, designed to be a direct replacement for Neighborhood Sampling with the same fanout hyperparameter while sampling much fewer vertices, without sacrificing quality.

### Hierarchical Estimation for Effective and Efficient Sampling Graph Neural Network

- Computer ScienceArXiv
- 2022

The proposed Hierarchical E stimation based S ampling GNN (HE-SGNN) with first level estimating the node embeddings in sampling probability to break circular dependency, and second level employing sampling GNN operator to estimate the nodes’representations on the entire graph.

### Not All Neighbors Are Worth Attending to: Graph Selective Attention Networks for Semi-supervised Learning

- Computer ScienceArXiv
- 2022

It is shown that a large portion of the neighbors are irrelevant to the central nodes in many real-world graphs, and can be excluded from neighbor aggregation.

### Hardware Acceleration of Sampling Algorithms in Sample and Aggregate Graph Neural Networks

- Computer Science
- 2022

A new neighbor sampler is proposed: CONCAT Sampler, which can be easily accel- erated on hardware level while guaranteeing the test accuracy, and a CONCAT-sampler-accelerator based on FPGA is designed, with which the neighbor sampling process boosted to about 300-1000 times faster compared to the sampling process without it.

### Rethinking Efficiency and Redundancy in Training Large-scale Graphs

- Computer Science
- 2022

This paper proposes DropReef, a once-for-all method to detect and drop the redundancy in large-scale graphs once and for all, helping reduce the training time while ensuring no sacriﬁce in the model accuracy.

### PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance

- Computer ScienceICML
- 2022

PLATON is proposed, which captures the uncertainty of importance scores by upper confidence bound (UCB) of importance estimation and tends to retain them and explores their capacity for the weights with low importance scores but high uncertainty.

### GNNLab: a factored system for sample-based GNN training over GPUs

- Computer ScienceEuroSys
- 2022

GNNLab adopts a factored design for multiple GPUs, where each GPU is dedicated to the task of graph sampling or model training, and proposes a new pre-sampling based caching policy that takes both sampling algorithms and GNN datasets into account, and shows an efficient and robust caching performance.

### Bandit Sampling for Multiplex Networks

- Computer ScienceArXiv
- 2022

This work proposes an algorithm for scalable learning on multiplex networks with a large number of layers that improves on the recent layer sampling method of DEEPLEX in that the unsampled layers do not need to be trained, enabling further increases in efficiency.

### Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks

- Computer ScienceICLR
- 2022

A communication-efficient distributed GNN training technique named Learn Locally, Correct Globally (LLCG), which can significantly improve the efficiency without hurting the performance and rigorously analyzes the convergence of distributed methods with periodic model averaging for training GNNs.

## References

SHOWING 1-10 OF 41 REFERENCES

### Bandit Samplers for Training Graph Neural Networks

- Computer ScienceNeurIPS
- 2020

This paper forms the optimization of the sampling variance as an adversary bandit problem, where the rewards are related to the node embeddings and learned weights, and can vary constantly.

### Advancing GraphSAGE with A Data-Driven Node Sampling

- Computer ScienceArXiv
- 2019

This work proposes a new data-driven sampling approach to reason about the real-valued importance of a neighborhood by a non-linear regressor, and to use the value as a criterion for subsampling neighborhoods.

### Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks

- Computer ScienceKDD
- 2020

This paper proposes a decoupled variance reduction strategy that employs (approximate) gradient information to adaptively sample nodes with minimal variance, and explicitly reduces the variance introduced by embedding approximation.

### Adaptive Sampling Towards Fast Graph Representation Learning

- Computer ScienceNeurIPS
- 2018

This paper develops an adaptive layer-wise sampling method that is adaptive and applicable for explicit variance reduction, which enhances the training of GCNs and proposes a novel and economical approach to promote the message passing over distant nodes by applying skip connections.

### FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling

- Computer ScienceICLR
- 2018

Enhanced with importance sampling, FastGCN not only is efficient for training but also generalizes well for inference, and is orders of magnitude more efficient while predictions remain comparably accurate.

### GraphSAINT: Graph Sampling Based Inductive Learning Method

- Computer ScienceICLR
- 2020

GraphSAINT is proposed, a graph sampling based inductive learning method that improves training efficiency in a fundamentally different way and can decouple the sampling process from the forward and backward propagation of training, and extend GraphSAINT with other graph samplers and GCN variants.

### Graph Convolutional Neural Networks for Web-Scale Recommender Systems

- Computer ScienceKDD
- 2018

A novel method based on highly efficient random walks to structure the convolutions and a novel training strategy that relies on harder-and-harder training examples to improve robustness and convergence of the model are developed.

### Stochastic Training of Graph Convolutional Networks with Variance Reduction

- Computer ScienceICML
- 2018

Control variate based algorithms which allow sampling an arbitrarily small neighbor size are developed and a new theoretical guarantee for these algorithms to converge to a local optimum of GCN is proved.

### Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking

- Computer ScienceICLR
- 2018

Graph2Gauss is proposed - an approach that can efficiently learn versatile node embeddings on large scale (attributed) graphs that show strong performance on tasks such as link prediction and node classification and the benefits of modeling uncertainty are demonstrated.

### Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

- Computer ScienceNeurIPS
- 2019

A new effective sampling algorithm called LADIES is proposed, which outperforms the previous sampling methods regarding both time and memory and is shown to have better generalization accuracy than original full-batch GCN, due to its stochastic nature.