Dynamic file striping and data layout transformation on parallel system with fluctuating I/O workload

  title={Dynamic file striping and data layout transformation on parallel system with fluctuating I/O workload},
  author={S. Son and S. Sehrish and W. Liao and R. Oldfield and A. Choudhary},
  journal={2013 IEEE International Conference on Cluster Computing (CLUSTER)},
As the number of compute cores on modern parallel machines increases to more than hundreds of thousands, scalable and consistent I/O performance is becoming hard to obtain due to fluctuating file system performance. This fluctuation is often caused by rebuilding RAID disk from hardware failures or concurrent jobs competing for I/O. We present a mechanism that stripes across a dynamically-selected subset of I/O servers with the lightest workload to achieve the best I/O bandwidth available from… Expand
2 Citations
Efficient task-local I/O operations of massively parallel applications
  • 3
  • Highly Influenced


Scalable massively parallel I/O to task-local files
  • 86
  • PDF
Using Subfiling to Improve Programming Flexibility and Performance of Parallel Shared-file I/O
  • 26
  • PDF
Dynamically adapting file domain partitioning methods for collective I/O based on underlying parallel file system locking protocols
  • W. Liao, A. Choudhary
  • Computer Science
  • 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis
  • 2008
  • 57
  • PDF
PLFS: a checkpoint filesystem for parallel applications
  • 308
  • PDF
Exploiting Lustre File Joining for Effective Collective IO
  • 87
  • PDF
Disk-directed I/O for MIMD multiprocessors
  • D. Kotz
  • Computer Science
  • OSDI '94
  • 1994
  • 368
  • PDF
Improving MPI-IO output performance with active buffering plus threads
  • 69
Scalable Design and Implementations for MPI Parallel Overlapping I/O
  • 15
ParColl: Partitioned Collective I/O on the Cray XT
  • Weikuan Yu, J. Vetter
  • Computer Science
  • 2008 37th International Conference on Parallel Processing
  • 2008
  • 35
  • PDF
Server-Directed Collective I/O in Panda
  • 206