A Plugin for HDF5 Using PLFS for Improved I/O Performance and Semantic Analysis


HDF5 is a data model, library and file format for storing and managing data. It is designed for flexible and efficient I/O for high volume and complex data. Natively, it uses a single-file format where multiple HDF5 objects are stored in a single file. In a parallel HDF5 application, multiple processes access a single file, thereby resulting in a performance bottleneck in I/O. Additionally, a single-file format does not allow semantic post processing on individual objects outside the scope of the HDF5 application. We have developed a new plugin for HDF5 using its Virtual Object Layer that serves two purposes: 1) it uses PLFS to convert the single-file layout into a data layout that is optimized for the underlying file system, and 2) it stores data in a unique way that enables semantic post-processing on data. We measure the performance of the plugin and discuss work leveraging the new semantic post-processing functionality enabled. We further discuss the applicability of this approach for exascale burst buffer storage systems.

DOI: 10.1109/SC.Companion.2012.102

Extracted Key Phrases

5 Figures and Tables

Cite this paper

@article{Mehta2012APF, title={A Plugin for HDF5 Using PLFS for Improved I/O Performance and Semantic Analysis}, author={Kshitij Mehta and John Bent and Aaron Torres and Gary Grider and Edgar Gabriel}, journal={2012 SC Companion: High Performance Computing, Networking Storage and Analysis}, year={2012}, pages={746-752} }