• Corpus ID: 232092233

A Biased Graph Neural Network Sampler with Near-Optimal Regret

@inproceedings{Zhang2021ABG,
  title={A Biased Graph Neural Network Sampler with Near-Optimal Regret},
  author={Qingru Zhang and David Paul Wipf and Quan Gan and Le Song},
  booktitle={Neural Information Processing Systems},
  year={2021}
}
Graph neural networks (GNN) have recently emerged as a vehicle for applying deep network architectures to graph and relational data. However, given the increasing size of industrial datasets, in many practical situations the message passing computations required for sharing information across GNN layers are no longer scalable. Although various sampling methods have been introduced to approximate full-graph training within a tractable budget, there remain unresolved complications such as high… 

Figures and Tables from this paper

Graph Belief Propagation Networks

This work introduces a model that combines the advantages of these two approaches, where the marginal probabilities in a conditional random field, similar to collective classification, and the potentials in the random field are learned through end-to-end training, akin to graph neural networks.

(LA)yer-neigh(BOR) Sampling: Defusing Neighborhood Explosion in GNNs

A new sampling algorithm called LAyer-neighBOR sampling (LABOR) is proposed, designed to be a direct replacement for Neighborhood Sampling with the same fanout hyperparameter while sampling much fewer vertices, without sacrificing quality.

Hierarchical Estimation for Effective and Efficient Sampling Graph Neural Network

The proposed Hierarchical E stimation based S ampling GNN (HE-SGNN) with first level estimating the node embeddings in sampling probability to break circular dependency, and second level employing sampling GNN operator to estimate the nodes’representations on the entire graph.

Not All Neighbors Are Worth Attending to: Graph Selective Attention Networks for Semi-supervised Learning

It is shown that a large portion of the neighbors are irrelevant to the central nodes in many real-world graphs, and can be excluded from neighbor aggregation.

Hardware Acceleration of Sampling Algorithms in Sample and Aggregate Graph Neural Networks

A new neighbor sampler is proposed: CONCAT Sampler, which can be easily accel- erated on hardware level while guaranteeing the test accuracy, and a CONCAT-sampler-accelerator based on FPGA is designed, with which the neighbor sampling process boosted to about 300-1000 times faster compared to the sampling process without it.

Rethinking Efficiency and Redundancy in Training Large-scale Graphs

This paper proposes DropReef, a once-for-all method to detect and drop the redundancy in large-scale graphs once and for all, helping reduce the training time while ensuring no sacrifice in the model accuracy.

PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance

PLATON is proposed, which captures the uncertainty of importance scores by upper confidence bound (UCB) of importance estimation and tends to retain them and explores their capacity for the weights with low importance scores but high uncertainty.

GNNLab: a factored system for sample-based GNN training over GPUs

GNNLab adopts a factored design for multiple GPUs, where each GPU is dedicated to the task of graph sampling or model training, and proposes a new pre-sampling based caching policy that takes both sampling algorithms and GNN datasets into account, and shows an efficient and robust caching performance.

Bandit Sampling for Multiplex Networks

This work proposes an algorithm for scalable learning on multiplex networks with a large number of layers that improves on the recent layer sampling method of DEEPLEX in that the unsampled layers do not need to be trained, enabling further increases in efficiency.

Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks

A communication-efficient distributed GNN training technique named Learn Locally, Correct Globally (LLCG), which can significantly improve the efficiency without hurting the performance and rigorously analyzes the convergence of distributed methods with periodic model averaging for training GNNs.

References

SHOWING 1-10 OF 41 REFERENCES

Bandit Samplers for Training Graph Neural Networks

This paper forms the optimization of the sampling variance as an adversary bandit problem, where the rewards are related to the node embeddings and learned weights, and can vary constantly.

Advancing GraphSAGE with A Data-Driven Node Sampling

This work proposes a new data-driven sampling approach to reason about the real-valued importance of a neighborhood by a non-linear regressor, and to use the value as a criterion for subsampling neighborhoods.

Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks

This paper proposes a decoupled variance reduction strategy that employs (approximate) gradient information to adaptively sample nodes with minimal variance, and explicitly reduces the variance introduced by embedding approximation.

Adaptive Sampling Towards Fast Graph Representation Learning

This paper develops an adaptive layer-wise sampling method that is adaptive and applicable for explicit variance reduction, which enhances the training of GCNs and proposes a novel and economical approach to promote the message passing over distant nodes by applying skip connections.

FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling

Enhanced with importance sampling, FastGCN not only is efficient for training but also generalizes well for inference, and is orders of magnitude more efficient while predictions remain comparably accurate.

GraphSAINT: Graph Sampling Based Inductive Learning Method

GraphSAINT is proposed, a graph sampling based inductive learning method that improves training efficiency in a fundamentally different way and can decouple the sampling process from the forward and backward propagation of training, and extend GraphSAINT with other graph samplers and GCN variants.

Graph Convolutional Neural Networks for Web-Scale Recommender Systems

A novel method based on highly efficient random walks to structure the convolutions and a novel training strategy that relies on harder-and-harder training examples to improve robustness and convergence of the model are developed.

Stochastic Training of Graph Convolutional Networks with Variance Reduction

Control variate based algorithms which allow sampling an arbitrarily small neighbor size are developed and a new theoretical guarantee for these algorithms to converge to a local optimum of GCN is proved.

Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking

Graph2Gauss is proposed - an approach that can efficiently learn versatile node embeddings on large scale (attributed) graphs that show strong performance on tasks such as link prediction and node classification and the benefits of modeling uncertainty are demonstrated.

Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks

A new effective sampling algorithm called LADIES is proposed, which outperforms the previous sampling methods regarding both time and memory and is shown to have better generalization accuracy than original full-batch GCN, due to its stochastic nature.