# xDGP: A Dynamic Graph Processing System with Adaptive Partitioning

@article{Vaquero2013xDGPAD, title={xDGP: A Dynamic Graph Processing System with Adaptive Partitioning}, author={Luis M. Vaquero and F{\'e}lix Cuadrado and Dionysios Logothetis and Claudio Martella}, journal={ArXiv}, year={2013}, volume={abs/1309.1049} }

Many real-world systems, such as social networks, rely on mining efficiently large graphs, with hundreds of millions of vertices and edges. This volume of information requires partitioning the graph across multiple nodes in a distributed system. This has a deep effect on performance, as traversing edges cut between partitions incurs a significant performance penalty due to the cost of communication. Thus, several systems in the literature have attempted to improve computational performance by…

## Figures and Tables from this paper

## 32 Citations

Planar: Parallel lightweight architecture-aware adaptive graph repartitioning

- Computer Science2016 IEEE 32nd International Conference on Data Engineering (ICDE)
- 2016

Planar is introduced, a parallel lightweight graph repartitioner, which does not require full knowledge of the graph and incrementally adapts the partitioning to changes while considering the heterogeneity and contentiousness of the underlying computing infrastructure.

GraphSteal: Dynamic Re-Partitioning for Efficient Graph Processing in Heterogeneous Clusters

- Computer Science2017 IEEE 10th International Conference on Cloud Computing (CLOUD)
- 2017

GraphSteal, a dynamic graph re-partitioning policy for vertex-cut based graph processing frameworks on heterogeneous clusters, is proposed and Experimental results show that GraphSteal significantly improves the performance over Graphlab.

PARAGON: Parallel Architecture-Aware Graph Partition Refinement Algorithm

- Computer ScienceEDBT
- 2016

PARAGON, a parallel architecture-aware graph partition refinement algorithm, which mitigates the mismatch by modifying a given decomposition according to the nonuniform network communication costs and the contentiousness of the underlying hardware topology, is introduced.

Architecture- and Workload-Aware Graph (Re)Partitioning

- Computer Science
- 2017

It is found that putting too much data communication into partitions assigned to cores of the same machines may result in serious contention for the shared hardware resources on the memory subsystems in modern multicore clusters.

SDP: Scalable Real-time Dynamic Graph Partitioner

- Computer ScienceIEEE Transactions on Services Computing
- 2021

The SDP contributes a novel vertex assigning method, communication-aware balancing method, and a scaling technique to produce an efficient dynamic graph partitioning method called Scalable Dynamic Graph Partitioner using the streaming partitioning technique.

Complex Network Partitioning Using Label Propagation

- Computer ScienceSIAM J. Sci. Comput.
- 2016

We present PuLP (partitioning using label propagation), a parallel and memory-efficient graph partitioning method specifically designed to partition low-diameter networks with skewed degree…

A Workload-Adaptive Streaming Partitioner for Distributed Graph Stores

- Computer ScienceData Sci. Eng.
- 2021

A novel workload-adaptive streaming partitioner named WASP, that aims to achieve low-latency and high-throughput online graph queries and achieves a better query performance against state-of-the-art graph partitioners, especially in dynamic query workloads.

PuLP: Scalable multi-objective multi-constraint partitioning for small-world networks

- Computer Science2014 IEEE International Conference on Big Data (Big Data)
- 2014

PuLP (Partitioning using Label Propagation) optimizes for multiple objective metrics simultaneously, while satisfying multiple partitioning constraints, and achieves better partitioning quality results for the multi-objective scenario.

An analysis of the graph processing landscape

- Computer ScienceJ. Big Data
- 2021

An overview of different aspects of the graph processing landscape is provided, different types of systems to use, coordination and communication models in distributed graph processing, partitioning techniques and different definitions related to the potential for a graph to be updated are described.

Scalable Graph Processing Frameworks

- Computer ScienceACM Comput. Surv.
- 2018

A taxonomy of graph processing systems is proposed and existing systems are mapped to this classification, which captures the diversity in programming and computation models, runtime aspects of partitioning and communication, both for in-memory and distributed frameworks.

## References

SHOWING 1-10 OF 46 REFERENCES

Streaming graph partitioning for large distributed graphs

- Computer ScienceKDD
- 2012

This work proposes natural, simple heuristics for graph partitioning and compares their performance to hashing and METIS, a fast, offline heuristic, and shows on a large collection of graph datasets that they are a significant improvement.

Towards effective partition management for large graphs

- Computer ScienceSIGMOD Conference
- 2012

This work proposes a Self Evolving Distributed Graph Management Environment (Sedge), to minimize inter-machine communication during graph query processing in multiple machines and introduces a two-level partition management architecture with complimentary primary partitions and dynamic secondary partitions.

Mizan: a system for dynamic load balancing in large-scale graph processing

- Computer ScienceEuroSys '13
- 2013

Mizan is introduced, a Pregel system that achieves efficient load balancing to better adapt to changes in computing needs and does not assume any a priori knowledge of the structure of the graph or behavior of the algorithm.

GPS: a graph processing system

- Computer ScienceSSDBM
- 2013

This paper describes the implementation of GPS and its novel features, and presents experimental results on the performance effects of both static and dynamic graph partitioning schemes, and describes the compilation of a high-level domain-specific programming language to GPS, enabling easy expression of complex algorithms.

Managing Large Graphs on Multi-Cores with Graph Awareness

- Computer ScienceUSENIX Annual Technical Conference
- 2012

Grace is a graph-aware, in-memory, transactional graph management system, specifically built for real-time queries and fast iterative computations, designed to run on large multi-cores to improve its performance.

A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs

- Computer ScienceSIAM J. Sci. Comput.
- 1998

This work presents a new coarsening heuristic (called heavy-edge heuristic) for which the size of the partition of the coarse graph is within a small factor of theSize of the final partition obtained after multilevel refinement, and presents a much faster variation of the Kernighan--Lin (KL) algorithm for refining during uncoarsening.

Components in time-varying graphs

- Computer Science, MathematicsChaos
- 2012

The notion of connectedness, and the definitions of node and graph components, are extended to the case of time-varying graphs, which are represented as time-ordered sequences of graphs defined over a fixed set of nodes.

Signal/Collect: Graph Algorithms for the (Semantic) Web

- Computer ScienceSEMWEB
- 2010

This paper presents the Signal/Collect programming model for synchronous and asynchronous graph algorithms and demonstrates that this abstraction can capture the essence of many algorithms on graphs in a concise and elegant way by giving Signal/ collect adaptations of various relevant algorithms.

Computational Methods for Dynamic Graphs

- Computer Science, Mathematics
- 2003

A data structure and an updating scheme are introduced that captures, in an approximate sense, the graph and its evolution through time and can be enhanced to form what the authors call communities of interest (COI).

Fast Random Walk with Restart and Its Applications

- Computer ScienceSixth International Conference on Data Mining (ICDM'06)
- 2006

The heart of the approach is to exploit two important properties shared by many real graphs: linear correlations and block- wise, community-like structure and exploit the linearity by using low-rank matrix approximation, and the community structure by graph partitioning, followed by the Sherman- Morrison lemma for matrix inversion.