Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system

@article{Kougkas2018HermesAH,
  title={Hermes: a heterogeneous-aware multi-tiered distributed I/O buffering system},
  author={Anthony Kougkas and Hariharan Devarajan and Xian-he Sun},
  journal={Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing},
  year={2018}
}
Modern High-Performance Computing (HPC) systems are adding extra layers to the memory and storage hierarchy named deep memory and storage hierarchy (DMSH), to increase I/O performance. [] Key Method Hermes enables, manages, supervises, and, in some sense, extends I/O buffering to fully integrate into the DMSH.
View on ACM
dl.acm.org
38 Citations
Highly Influential Citations
1
Background Citations
25
Methods Citations
6

38 Citations

Citation Type

Citation Type

UniviStor: Integrated Hierarchical and Distributed Storage for HPC
TLDR
UniviStor is introduced, a data management service offering a unified view of storage layers that provides performance optimizations and data structures tailored for distributed and hierarchical data placement, interferenceaware data movement scheduling, adaptive data striping, and lightweight workflow management.
HFlow: A Dynamic and Elastic Multi-Layered I/O Forwarder
TLDR
HFlow is presented, a new class of data forwarding system that leverages a real-time data movement paradigm providing data-independent tasks that can be executed anywhere and thus, enabling dynamic resource provisioning and shows an increase in performance of 3x when compared with state-of-the-art software solutions.
Harmonia: An Interference-Aware Dynamic I/O Scheduler for Shared Non-volatile Burst Buffers
TLDR
Harmonia is introduced, a new dynamic I/O scheduler that is aware of interference, adapts to the underlying system, implements a new 2-way decision-making process and employs several scheduling policies to maximize the system efficiency and applications' performance.
HCompress: Hierarchical Data Compression for Multi-Tiered Storage Environments
TLDR
HCompress is a hierarchical data compression library that can improve the application’s performance by harmoniously leveraging both multi-tiered storage and data compression, and has been developed a novel compression selection algorithm that facilitates the optimal matching of compression libraries to the tiered storage.
MLBS: Transparent Data Caching in Hierarchical Storage for Out-of-Core HPC Applications
TLDR
MultiLayered Buffer Storage (MLBS), a data object container that provides novel methods for caching and prefetching data in out-of-core scientific applications to perform asynchronously expensive I/O operations on systems equipped with hierarchical storage, is introduced.
Interference-Aware I/O Scheduling for Data-Intensive Applications on Hierarchical HPC Storage Systems
  • Weihao Liang, Yong Chen, Hong An
  • Computer Science
    2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS)
  • 2019
TLDR
This paper presents a dynamic, interference-aware scheduling scheme that can efficiently manages the I/O scheduling among different layers of hierarchical HPC storage system to coordinate multiple concurrent data-intensive applications.
Maximizing I/O Bandwidth for Reverse Time Migration on Heterogeneous Large-Scale Systems
TLDR
An extension to the Multilayer Buffer System framework to further maximize RTM I/O bandwidth in presence of GPU hardware accelerators and to leverage the GPU’s High Bandwidth Memory (HBM) as an additional storage media layer is introduced.
Bridging Storage Semantics Using Data Labels and Asynchronous I/O
TLDR
The idea of a Label, a new data representation, and, LABIOS: a new, distributed, Label- based I/O system are introduced, which demonstrates the effectiveness of storage bridging to support the convergence of HPC and BigData workloads on a single platform.
A Dynamic Multi-Tiered Storage System for Extreme Scale Computing
TLDR
The I/O bottleneck problem presents a significant challenge for scientific applications on modern supercomputers that must be solved by the next generation of storage systems.
Efficient Data Eviction across Multiple Tiers of Storage
TLDR
RFlush is a real-time data flushing platform for multi-tiered storage environments that allows RFlush to provide a low latency and autoscaling capabilities while also providing an efficient pipeline for continuous dataFlushing operations to enable high resource utilization.
...
1
2
3
4
...

References

SHOWING 1-10 OF 72 REFERENCES
Harmonia: An Interference-Aware Dynamic I/O Scheduler for Shared Non-volatile Burst Buffers
TLDR
Harmonia is introduced, a new dynamic I/O scheduler that is aware of interference, adapts to the underlying system, implements a new 2-way decision-making process and employs several scheduling policies to maximize the system efficiency and applications' performance.
Data Elevator: Low-Contention Data Movement in Hierarchical Storage System
  • Bin Dong, S. Byna, N. Keen
  • Computer Science
    2016 IEEE 23rd International Conference on High Performance Computing (HiPC)
  • 2016
TLDR
This paper proposes a new system, named Data Elevator, for transparently and efficiently moving data in hierarchical storage, which reduces the resource contention on BB servers via offloading the data movement from a fixed number of BB server nodes to compute nodes.
Using active NVRAM for I/O staging
TLDR
This paper proposes a mechanism, in which each physical node has an additional active NVRAM component to stage I/O and apply simple data analytics operations over theI/O data, and experimental results show the effectiveness of the approach in addressing 'right memory sizing issue' by efficient I-O data processing.
Unimem: runtime data managementon non-volatile memory-based heterogeneous main memory
TLDR
It is demonstrated that using NVM to replace the majority of DRAM can be a feasible solution for future HPC systems with the assistance of a software-based data management.
Improving MPI-IO output performance with active buffering plus threads
TLDR
This work shows that the implementation of ABT in ROMIO, a popular implementation of MPI-IO, greatly reduces the application-visible cost of ROMIO's collective write calls, and improves an application's overall performance by hiding I/O cost and saving implicit synchronization overhead from collective write operations.
IO-Lite: a unified I/O buffering and caching system
TLDR
IO-Lite eliminates all copying and multiple buffering of I/O data, and enables various cross-subsystem optimizations, and shows performance improvements between 40 and 80% on real workloads as a result of IO -Lite.
Moneta: A High-Performance Storage Array Architecture for Next-Generation, Non-volatile Memories
TLDR
The architecture of a prototype PCIe-attached storage array built from emulated PCM storage called Moneta, which provides a carefully designed hardware/software interface that makes issuing and completing accesses atomic and explores trade-offs in Moneta's architecture between performance, power, memory organization, and memory latency.
Gordon: using flash memory to build fast, power-efficient clusters for data-intensive applications
TLDR
The paper presents an exhaustive analysis of the design space of Gordon systems, focusing on the trade-offs between power, energy, and performance that Gordon must make, and describes a novel flash translation layer tailored to data intensive workloads and large flash storage arrays.
Collective buffering: Improving parallel I/O performance
  • B. Nitzberg, V. Lo
  • Computer Science
    Proceedings. The Sixth IEEE International Symposium on High Performance Distributed Computing (Cat. No.97TB100183)
  • 1997
TLDR
The general model of the problem is discussed, four Collective Buffering algorithms are described, and experiments show improvements of up to two order of magnitude over standard techniques and the potential to deliver peak performance with minimal hardware support.
A User-Level InfiniBand-Based File System and Checkpoint Strategy for Burst Buffers
  • Kento Sato, K. Mohror, S. Matsuoka
  • Computer Science, Business
    2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing
  • 2014
TLDR
A user-level Infini Band-based file system (IBIO) is developed that exploits the bandwidth of burst buffers, and performance models for coordinated and uncoordinated checkpoint/restart strategies are developed and applied to investigate the best checkpoint strategy using burst buffers on future large-scale systems.
...
1
2
3
4
5
...