• Corpus ID: 231662123

Learning Massive Graph Embeddings on a Single Machine

@inproceedings{Mohoney2021LearningMG,
  title={Learning Massive Graph Embeddings on a Single Machine},
  author={Jason Mohoney and Roger Waleffe and Yiheng Xu and Theodoros Rekatsinas and Shivaram Venkataraman},
  booktitle={OSDI},
  year={2021}
}
We propose a new framework for computing the embeddings of large-scale graphs on a single machine. A graph embedding is a fixed length vector representation for each node (and/or edge-type) in a graph and has emerged as the de-facto approach to apply modern machine learning on graphs. We identify that current systems for learning the embeddings of large-scale graphs are bottlenecked by data movement, which results in poor resource utilization and inefficient training. These limitations require… 
SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs
TLDR
Scalable Multi-hOp REasoning (SMORE), the first general framework for both single-hop and multi-hop reasoning in KGs, is presented and achieves comparable or even better runtime performance to state-of-the-art frameworks on both single GPU andMulti-GPU settings.
Marius++: Large-Scale Training of Graph Neural Networks on a Single Machine
TLDR
It is shown that out-of-core pipelined mini-batch training in a single machine outperforms resource-hungry multi-GPU solutions and Marius++, a system for training GNNs over billion-scale graphs, is introduced.
Demo of Marius: A System for Large-scale Graph Embeddings
A Birds Eye View on Knowledge Graph Embeddings, Software Libraries, Applications and Challenges
TLDR
Existing KGC approaches are discussed, including the state-of-the-art Knowledge Graph Embeddings (KGE), not only on static graphs but also for the latest trends such as multimodal, temporal, and uncertain knowledge graphs.
Algorithm and System Co-design for Efficient Subgraph-based Graph Representation Learning
TLDR
A novel framework SUREL is proposed for scalable SGRL by co-designing the learning algorithm and its system support and significantly outperforms the state-of-the-art baseline in both prediction accuracy and efficiency.
Fograph: Enabling Real-Time Deep Graph Inference with Fog Computing
TLDR
Fograph is presented, a novel distributed real-time GNN inference framework that leverages diverse resources of multiple fog nodes in proximity to IoT data sources that significantly outperforms the state-of-the-art cloud serving and vanilla fog deployment.
Machine Learning and Data Cleaning: Which Serves the Other?
TLDR
This symbiotic relationship between ML and data cleaning is highlighted and few challenges that require collaborative eforts of multiple research communities are discussed.
Saga: A Platform for Continuous Construction and Serving of Knowledge At Scale
TLDR
The unique challenges associated with knowledge graph construction at industrial scale are discussed, and the main components of Saga are reviewed and how they address these challenges are reviewed.
SmartSAGE: Training Large-scale Graph Neural Networks using In-Storage Processing Architectures
TLDR
SmartSAGE, a software/hardware co-design based on an in-storage processing (ISP) architecture, demonstrates that an ISP based large-scale GNN training system can achieve both high capacity storage and high performance, opening up opportunities for ML practitioners to train large GNN datasets without being hampered by the physical limitations of main memory size.
Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards
TLDR
The proposed ScratchPipe architecture utilizes unique properties of RecSys training to develop an embedding cache that not only sees the past but also the “future” cache accesses, enabling embedding layer training to be conducted at GPU memory speed.
...
1
2
...

References

SHOWING 1-10 OF 48 REFERENCES
PyTorch-BigGraph: A Large-scale Graph Embedding System
TLDR
PyTorch-BigGraph (PBG), an embedding system that incorporates several modifications to traditional multi-relation embedding systems that allow it to scale to graphs with billions of nodes and trillions of edges, is presented.
GraphChi: Large-Scale Graph Computation on Just a PC
TLDR
This work presents GraphChi, a disk-based system for computing efficiently on graphs with billions of edges, and builds on the basis of Parallel Sliding Windows to propose a new data structure Partitioned Adjacency Lists, which is used to design an online graph database graphChi-DB.
DGL-KE: Training Knowledge Graph Embeddings at Scale
TLDR
DGL-KE introduces various novel optimizations that accelerate training on knowledge graphs with millions of nodes and billions of edges using multi-processing, multi-GPU, and distributed parallelism to increase data locality, reduce communication overhead, overlap computations with memory accesses, and achieve high operation efficiency.
Complex Embeddings for Simple Link Prediction
TLDR
This work makes use of complex valued embeddings to solve the link prediction problem through latent factorization, and uses the Hermitian dot product, the complex counterpart of the standard dot product between real vectors.
Embedding Entities and Relations for Learning and Inference in Knowledge Bases
TLDR
It is found that embeddings learned from the bilinear objective are particularly good at capturing relational semantics and that the composition of relations is characterized by matrix multiplication.
Scalability! But at what COST?
TLDR
This work surveys measurements of data-parallel systems recently reported in SOSP and OSDI, and finds that many systems have either a surprisingly large COST, often hundreds of cores, or simply underperform one thread for all of their reported configurations.
The impossibility of low-rank representations for triangle-rich complex networks
TLDR
It is mathematically proved that low-dimensional embeddings cannot generate graphs with both low average degree and large clustering coefficients, and any embedding that can successfully create these two properties must have a rank that is nearly linear in the number of vertices.
A Capsule Network-based Embedding Model for Knowledge Graph Completion and Search Personalization
TLDR
The proposed CapsE obtains better performance than previous state-of-the-art embedding models for knowledge graph completion on two benchmark datasets WN18RR and FB15k-237, and outperforms strong search personalization baselines on SEARCH17.
GraphVite: A High-Performance CPU-GPU Hybrid System for Node Embedding
TLDR
This paper proposes GraphVite, a high-performance CPU-GPU hybrid system for training node embeddings, by co-optimizing the algorithm and the system, and proposes an efficient collaboration strategy to further reduce the synchronization cost between CPUs and GPUs.
Optimizing data-intensive computations in existing libraries with split annotations
TLDR
Mozart is implemented, a new technique called split annotations (SAs) that enables key data movement optimizations over unmodified library functions and provides performance gains competitive with solutions that require rewriting libraries, and can sometimes outperform these systems by up to 2x by leveraging existing hand-optimized code.
...
1
2
3
4
5
...