Scalable stream join processing with expensive predicates: workload distribution and adaptation by time-slicing

Abstract

Multi-way stream joins with expensive join predicates lead to great challenge for real-time (or close to real-time) stream processing. Given the memory- and CPU-intensive nature of such stream join queries, scalable processing on a cluster must be employed. This paper proposes a novel scheme for distributed processing of generic multi-way joins with window constraints, called Pipelined State Partitioning (PSP). We target generic joins with arbitrarily join conditions, which are used in non-trivial stream applications such as image matching and biometric recognizing. The PSP scheme partitions the states into disjoint slices in the time domain, and then distributes the fine-grained states in the cluster, forming a virtual computation ring. Compared to replication-based distribution of non-equi-joins, PSP scheme is superior since: (1) zero state duplication and thus no repeated computations, (2) pipelined processing of every input tuple on multiple nodes to achieve low response time, and (3) cost-based adaptive workload distribution. We have implemented the proposed PSP schemes within the CAPE DSMS. Our experimental study demonstrates the significant performance improvements compared to the state-of-the-art generic distributed stream join algorithms.

DOI: 10.1145/1516360.1516396

Extracted Key Phrases

8 Figures and Tables

Cite this paper

@inproceedings{Wang2009ScalableSJ, title={Scalable stream join processing with expensive predicates: workload distribution and adaptation by time-slicing}, author={Song Wang and Elke A. Rundensteiner}, booktitle={EDBT}, year={2009} }