Dynamic file striping and data layout transformation on parallel system with fluctuating I/O workload

@article{Son2013DynamicFS,
  title={Dynamic file striping and data layout transformation on parallel system with fluctuating I/O workload},
  author={Seung Woo Son and Saba Sehrish and Wei-keng Liao and Ron Oldfield and Alok N. Choudhary},
  journal={2013 IEEE International Conference on Cluster Computing (CLUSTER)},
  year={2013},
  pages={1-8}
}
As the number of compute cores on modern parallel machines increases to more than hundreds of thousands, scalable and consistent I/O performance is becoming hard to obtain due to fluctuating file system performance. This fluctuation is often caused by rebuilding RAID disk from hardware failures or concurrent jobs competing for I/O. We present a mechanism that stripes across a dynamically-selected subset of I/O servers with the lightest workload to achieve the best I/O bandwidth available from… CONTINUE READING

Citations

Publications citing this paper.

Efficient task-local I/O operations of massively parallel applications

VIEW 14 EXCERPTS
CITES BACKGROUND & METHODS
HIGHLY INFLUENCED

References

Publications referenced by this paper.
SHOWING 1-10 OF 35 REFERENCES

PLFS: a checkpoint filesystem for parallel applications

VIEW 5 EXCERPTS
HIGHLY INFLUENTIAL

Characterizing output bottlenecks in a supercomputer

VIEW 5 EXCERPTS
HIGHLY INFLUENTIAL

Managing Variability in the IO Performance of Petascale Storage Systems

VIEW 5 EXCERPTS

A Case Study for Scientific I/O: Improving the FLASH Astrophysics Code

  • R. Latham
  • Computational Science & Discovery, vol. 5, no. 1, 2012.
  • 2012
VIEW 2 EXCERPTS

Efficient data restructuring and aggregation for I/O acceleration in PIDX

VIEW 1 EXCERPT

PIDX: Efficient Parallel I/O for Multi-resolution Multi-dimensional Scientific Datasets

VIEW 1 EXCERPT

Parallel I/O Performance for Application-Level Checkpointing on the Blue Gene/P System

VIEW 1 EXCERPT