Social Hash Partitioner: A Scalable Distributed Hypergraph Partitioner

  title={Social Hash Partitioner: A Scalable Distributed Hypergraph Partitioner},
  author={Igor Kabiljo and Brian Karrer and Mayank Pundir and Sergey Pupyrev and Alon Shalita and Alessandro Presta and Yaroslav Akhremtsev},
  journal={Proc. VLDB Endow.},
We design and implement a distributed algorithm for balanced k-way hypergraph partitioning that minimizes fanout, a fundamental hypergraph quantity also known as the communication volume and (k - 1)-cut metric, by optimizing a novel objective called probabilistic fanout. This choice allows a simple local search heuristic to achieve comparable solution quality to the best existing hypergraph partitioners. Our algorithm is arbitrarily scalable due to a careful design that controls… 

Figures and Tables from this paper

Shared-Memory n-level Hypergraph Partitioning

A shared-memory algorithm to compute high-quality solutions to the balanced k-way hypergraph partitioning problem and shows that recent non-multilevel algorithms specifically designed to partition large instances have considerable quality penalties and no clear advantage in running time.

Prioritized Restreaming Algorithms for Balanced Graph Partitioning

This work observes that two recently introduced families of iterative partitioners---those based on restreaming and those based on balanced label propagation---can be viewed through a common modular framework of design decisions, and finds a novel family of algorithms with notably better empirical performance than any existing highly-scalable algorithm on a broad range of real-world graphs.

High-Quality Hypergraph Partitioning

This paper considers the fundamental and intensively studied problem of balanced hypergraph partitioning, which asks for partitioning the vertices into k disjoint blocks of bounded size while minimizing an objective function over the hyperedges.

HYPE: Massive Hypergraph Partitioning with Neighborhood Expansion

HYPE is proposed, a hypergraph partitionier that exploits the neighborhood relations between vertices in the hypergraph using an efficient implementation of neighborhood expansion and improves partitioning quality and reduces runtime compared to streaming partitioning.

Advanced Flow-Based Multilevel Hypergraph Partitioning

The recently proposed HyperFlowCutter algorithm for computing bipartitions of unweighted hypergraphs by solving a sequence of incremental maximum flow problems is enhanced to handle weighted instances and a technique for computing maximum flows directly on weighted hyper graphs is proposed.

Scalable Shared-Memory Hypergraph Partitioning

Mt-KaHyPar is the first shared-memory multilevel hypergraph partitioner with parallel implementations of many techniques used by the sequential, high-quality partitioning systems: a parallel coarsening algorithm that uses parallel community detection as guidance, initial partitioning via parallel recursive bipartitioning with work-stealing, a scalable label propagation refinement algorithm, and the first fully-parallel direct $k$-way formulation of the classical FM algorithm.

BiPart: a parallel and deterministic hypergraph partitioner

Experimental results show that BiPart outperforms state-of-the-art hypergraph partitioners in runtime and partition quality while generating partitions deterministically.

Asynchronous n-Level Hypergraph Partitioning

In this thesis, we introduce a novel approach to scalable high-quality parallel hypergraph partitioning. The balanced k-way hypergraph partitioning problem consists of partitioning the vertices of a

Distributed Edge Partitioning for Trillion-edge Graphs

It is theoretically prove that the proposed Distributed Neighbor Expansion method has the upper bound in the partitioning quality, and the performance evaluation shows that the space efficiency of the proposed method is an order-of-magnitude better than the existing algorithms, keeping its time efficiency comparable.

Parallel Flow-Based Hypergraph Partitioning

This work presents a shared-memory parallelization of flow-based refinement, which is considered the most powerful iterative improvement technique for hypergraph partitioning at the moment, and investigates two different sources of parallelism: a parallel scheduling scheme and a parallel maximum flow algorithm based on the well-known push-relabel algorithm.



A Distributed Algorithm for Balanced Hypergraph Partitioning

This paper proposes a distributed hyperedge partition algorithm, HyperSwap, to partition the hypergraph into balanced sub-hypergraph as required, without global information and central coordination, and shows the feasibility, evaluate it on Facebook dataset with various settings, and compare it against two alternative solutions.

Hypergraph partitioning for multiple communication cost metrics: Model and methods

Distributed data placement to minimize communication costs via graph partitioning

This work reduces the data placement problem to the well-studied problem of Graph Partitioning, which is NP-Hard but for which efficient approximation algorithms exist, and produces nearly-optimal solutions in seconds.

Replicated partitioning for undirected hypergraphs

Social Hash: An Assignment Framework for Optimizing Distributed Systems Operations on Social Networks

The framework uses a two-level scheme to decouple compute-intensive optimization from relatively low-overhead dynamic adaptation to optimize the operations of large social networks, such as Facebook's Social Graph.

Balanced Partitions of Trees and Applications

It is shown that the k-BALANCED PARTITIONING problem remains APX-hard even when restricted to unweighted tree instances with constant maximum degree, and it is proved that the problem is NP-hard to approximate within nc, for any constant c<1.

Parallel multilevel algorithms for hypergraph partitioning

Partitioning graphs into balanced components

This work considers the k-balanced partitioning problem, where the goal is to partition the vertices of an input graph G into k equally sized components, while minimizing the total weight of the edges connecting different components, and presents a (bi-criteria) approximation algorithm achieving an approximation of O(log n log k), which matches or improves over previous algorithms for all relevant values of k.

A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs

This work presents a new coarsening heuristic (called heavy-edge heuristic) for which the size of the partition of the coarse graph is within a small factor of theSize of the final partition obtained after multilevel refinement, and presents a much faster variation of the Kernighan--Lin (KL) algorithm for refining during uncoarsening.

Multilevel Hypergraph Partitioning: Application In Vlsi Domain

The experiments show that the multilevel hypergraph partitioning algorithm produces high quality partitioning in relatively small amount of time and outperforms other schemes (in hyperedge cut) quite consistently with larger margins.