• Corpus ID: 18020600

xDGP: A Dynamic Graph Processing System with Adaptive Partitioning

@article{Vaquero2013xDGPAD,
  title={xDGP: A Dynamic Graph Processing System with Adaptive Partitioning},
  author={Luis M. Vaquero and F{\'e}lix Cuadrado and Dionysios Logothetis and Claudio Martella},
  journal={ArXiv},
  year={2013},
  volume={abs/1309.1049}
}
Many real-world systems, such as social networks, rely on mining efficiently large graphs, with hundreds of millions of vertices and edges. This volume of information requires partitioning the graph across multiple nodes in a distributed system. This has a deep effect on performance, as traversing edges cut between partitions incurs a significant performance penalty due to the cost of communication. Thus, several systems in the literature have attempted to improve computational performance by… 

Figures and Tables from this paper

Planar: Parallel lightweight architecture-aware adaptive graph repartitioning
TLDR
Planar is introduced, a parallel lightweight graph repartitioner, which does not require full knowledge of the graph and incrementally adapts the partitioning to changes while considering the heterogeneity and contentiousness of the underlying computing infrastructure.
GraphSteal: Dynamic Re-Partitioning for Efficient Graph Processing in Heterogeneous Clusters
TLDR
GraphSteal, a dynamic graph re-partitioning policy for vertex-cut based graph processing frameworks on heterogeneous clusters, is proposed and Experimental results show that GraphSteal significantly improves the performance over Graphlab.
PARAGON: Parallel Architecture-Aware Graph Partition Refinement Algorithm
TLDR
PARAGON, a parallel architecture-aware graph partition refinement algorithm, which mitigates the mismatch by modifying a given decomposition according to the nonuniform network communication costs and the contentiousness of the underlying hardware topology, is introduced.
Architecture- and Workload-Aware Graph (Re)Partitioning
TLDR
It is found that putting too much data communication into partitions assigned to cores of the same machines may result in serious contention for the shared hardware resources on the memory subsystems in modern multicore clusters.
SDP: Scalable Real-time Dynamic Graph Partitioner
TLDR
The SDP contributes a novel vertex assigning method, communication-aware balancing method, and a scaling technique to produce an efficient dynamic graph partitioning method called Scalable Dynamic Graph Partitioner using the streaming partitioning technique.
Complex Network Partitioning Using Label Propagation
We present PuLP (partitioning using label propagation), a parallel and memory-efficient graph partitioning method specifically designed to partition low-diameter networks with skewed degree
A Workload-Adaptive Streaming Partitioner for Distributed Graph Stores
TLDR
A novel workload-adaptive streaming partitioner named WASP, that aims to achieve low-latency and high-throughput online graph queries and achieves a better query performance against state-of-the-art graph partitioners, especially in dynamic query workloads.
PuLP: Scalable multi-objective multi-constraint partitioning for small-world networks
TLDR
PuLP (Partitioning using Label Propagation) optimizes for multiple objective metrics simultaneously, while satisfying multiple partitioning constraints, and achieves better partitioning quality results for the multi-objective scenario.
An analysis of the graph processing landscape
TLDR
An overview of different aspects of the graph processing landscape is provided, different types of systems to use, coordination and communication models in distributed graph processing, partitioning techniques and different definitions related to the potential for a graph to be updated are described.
Scalable Graph Processing Frameworks
TLDR
A taxonomy of graph processing systems is proposed and existing systems are mapped to this classification, which captures the diversity in programming and computation models, runtime aspects of partitioning and communication, both for in-memory and distributed frameworks.
...
...

References

SHOWING 1-10 OF 46 REFERENCES
Streaming graph partitioning for large distributed graphs
TLDR
This work proposes natural, simple heuristics for graph partitioning and compares their performance to hashing and METIS, a fast, offline heuristic, and shows on a large collection of graph datasets that they are a significant improvement.
Towards effective partition management for large graphs
TLDR
This work proposes a Self Evolving Distributed Graph Management Environment (Sedge), to minimize inter-machine communication during graph query processing in multiple machines and introduces a two-level partition management architecture with complimentary primary partitions and dynamic secondary partitions.
Mizan: a system for dynamic load balancing in large-scale graph processing
TLDR
Mizan is introduced, a Pregel system that achieves efficient load balancing to better adapt to changes in computing needs and does not assume any a priori knowledge of the structure of the graph or behavior of the algorithm.
GPS: a graph processing system
TLDR
This paper describes the implementation of GPS and its novel features, and presents experimental results on the performance effects of both static and dynamic graph partitioning schemes, and describes the compilation of a high-level domain-specific programming language to GPS, enabling easy expression of complex algorithms.
Managing Large Graphs on Multi-Cores with Graph Awareness
TLDR
Grace is a graph-aware, in-memory, transactional graph management system, specifically built for real-time queries and fast iterative computations, designed to run on large multi-cores to improve its performance.
A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs
TLDR
This work presents a new coarsening heuristic (called heavy-edge heuristic) for which the size of the partition of the coarse graph is within a small factor of theSize of the final partition obtained after multilevel refinement, and presents a much faster variation of the Kernighan--Lin (KL) algorithm for refining during uncoarsening.
Components in time-varying graphs
TLDR
The notion of connectedness, and the definitions of node and graph components, are extended to the case of time-varying graphs, which are represented as time-ordered sequences of graphs defined over a fixed set of nodes.
Signal/Collect: Graph Algorithms for the (Semantic) Web
TLDR
This paper presents the Signal/Collect programming model for synchronous and asynchronous graph algorithms and demonstrates that this abstraction can capture the essence of many algorithms on graphs in a concise and elegant way by giving Signal/ collect adaptations of various relevant algorithms.
Computational Methods for Dynamic Graphs
TLDR
A data structure and an updating scheme are introduced that captures, in an approximate sense, the graph and its evolution through time and can be enhanced to form what the authors call communities of interest (COI).
Fast Random Walk with Restart and Its Applications
TLDR
The heart of the approach is to exploit two important properties shared by many real graphs: linear correlations and block- wise, community-like structure and exploit the linearity by using low-rank matrix approximation, and the community structure by graph partitioning, followed by the Sherman- Morrison lemma for matrix inversion.
...
...