The data cyclotron query processing scheme

  title={The data cyclotron query processing scheme},
  author={Romulo Goncalves and Martin L. Kersten},
  journal={ACM Trans. Database Syst.},
A grand challenge of distributed query processing is to devise a self-organizing architecture which exploits all hardware resources optimally to manage the database hot set, minimize query response time, and maximize throughput without single point global coordination. The Data Cyclotron architecture [Goncalves and Kersten 2010] addresses this challenge using turbulent data movement through a storage ring built from distributed main memory and capitalizing on the functionality offered by modern… 

Rack-Scale In-Memory Join Processing using RDMA

This paper focuses on implementing parallel in-memory joins using Remote Direct Memory Access (RDMA), a communication mechanism to transfer data directly into the memory of a remote machine and develops an analytical model and test the implementation on two different types of networks.

Just-In-Time Data Distribution for Analytical Query Processing

This paper explores an alternative approach that starts from a master node in control of the complete database, and a variable number of worker nodes for delegated query processing, and shows that the proposed adaptive distributed architecture is a viable and flexible alternative for small scale MapReduce-type of settings.

A Spinning Join That Does Not Get Dizzy

This work proposes Data Roundabout as alight weight transport layer that uses Remote Direct Memory Access (RDMA) to gain access to the throughput opportunities in modern networks and demonstrates cyclo-join, which exploits the cycling flow of data to execute distributed joins.

High-Speed Query Processing over High-Speed Networks

An extensive evaluation within the HyPer database system using the TPC-H benchmark shows that the holistic approach presented indeed enables high-speed query processing over high- speed networks.

Predictable performance and high query concurrency for data analytics

This paper presents Cjoin, a new design that substantially improves throughput in large-scale data analytics systems processing many concurrent join queries, and implemented as an extension to the PostgreSQL DBMS.

Dynamic Load Balancing in Data Grids by Global Load Estimation

This work presents a dynamic replication strategy which prevents hot spots by dynamically replicating the hot data on different locations by using an optimization problem to determine the data to be replicated and the optimal replica receivers.

Parallel join executions in RAMCloud

This paper provides a system model to derive the execution costs for the Grace Join, the Distributed Block Nested Loop Join, and the Cyclo Join algorithm and their corresponding implementations in RAMCloud, a DRAM-based storage system connected via RDMA-enabled network adapters.

Efficient Distributed Join Processing on Modern Hardware

This thesis describes three efficient distributed algorithms for joining two large distributed relations and shows that the performance of distributed join algorithms on today’s commodity hardware is bound by the underlying network bandwidth.

QbDJ: A Novel Framework for Handling Skew in Parallel Join Processing on Distributed Memory

  • Long ChengS. KotoulasT. WardG. Theodoropoulos
  • Computer Science
    2013 IEEE 10th International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing
  • 2013
This paper introduces a novel parallel join framework, query-based distributed join (QbDJ), for handling data skew on distributed architectures and presents an efficient implementation of the method based on the asynchronous partitioned global address space (APGAS) parallel programming model.

Flow-Join: Adaptive skew handling for distributed joins over high-speed networks

This work introduces Flow-Join, a novel distributed join algorithm that handles attribute value skew with minimal overhead, and uses its novel lightweight skew handling scheme to execute at the full network speed of more than 6 GB/s for InfiniBand 4×FDR.



A Spinning Join That Does Not Get Dizzy

This work proposes Data Roundabout as alight weight transport layer that uses Remote Direct Memory Access (RDMA) to gain access to the throughput opportunities in modern networks and demonstrates cyclo-join, which exploits the cycling flow of data to execute distributed joins.

Towards self-tuning data placement in parallel database systems

A new index-based method that faciliates fast and efficient migration of data in a shared nothing system that incorporates a globally height-balanced structure and load tracking at different levels of granularity is introduced.

The DataPath system: a data-centric analytic processing engine for large data warehouses

In DataPath, queries do not request data, and data are automatically pushed onto processors, where they are then processed by any interested computation, making for a very lean and fast database system.

The state of the art in distributed query processing

The paper presents the “textbook” architecture for distributed query processing and a series of techniques that are particularly useful for distributed database systems, and discusses different kinds of distributed systems such as client-server, middleware (multitier), and heterogeneous database systems and shows how query processing works in these systems.

Allocating Resources to Parallel Query Plans in Data Grids

A heuristic algorithm that takes care of both pipelining and partitioning mechanisms while optimizing the allocation of resources to the query sub-plans for parallel query plan scheduling problem in data grids.

A high-performance computing method for data allocation in distributed database systems

An integrated method for grouping the distributed sites into clusters and customizing the database fragments allocation to the clusters and their sites is designed and found it reduces the data transferred between the sites during the execution time, minimizes the communication cost needed for processing applications, and handles the database queries and meets their future needs.

Optimizing Main-Memory Join on Modern Hardware

The partitioned hash-join is refined with a new partitioning algorithm called radix-cluster, which is specifically designed to optimize memory access, and the effect of implementation techniques that optimize CPU resource usage is investigated.

Cooperative Scans: Dynamic Bandwidth Sharing in a DBMS

This paper analyzes the performance of concurrent (index) scan operations in both record (NSM/PAX) and column (DSM) disk storage models and proposes the Cooperative Scans framework that enhances performance in such scenarios by improving data-sharing between concurrent scans.

Spinning relations: high-speed networks for distributed join processing

By leveraging modern networking hardware (RDMA-enabled network cards), we can shift priorities in distributed database processing significantly. Complex and sophisticated mechanisms to avoid network

The Datacycle architecture

The Datacycle TM architecture is a radical approach to database management that attempts to achieve a full separation of concerns between the needs of many applications and the database management system supporting those needs, even for applications with unusually high demands for flexibility in access to data and for performance.