Corpus ID: 8788414

High Performance Multi-Node File Copies and Checksums for Clustered File Systems

@inproceedings{Kolano2010HighPM,
  title={High Performance Multi-Node File Copies and Checksums for Clustered File Systems},
  author={Paul Z. Kolano and R. Ciotti},
  booktitle={LISA},
  year={2010}
}
  • Paul Z. Kolano, R. Ciotti
  • Published in LISA 2010
  • Computer Science
  • Mcp and msum are drop-in replacements for the standard cp and md5sum programs that utilize multiple types of parallelism and other optimizations to achieve maximum copy and checksum performance on clustered file systems. Multi-threading is used to ensure that nodes are kept as busy as possible. Read/write parallelism allows individual operations of a single copy to be overlapped using asynchronous I/O. Multi-node cooperation allows different nodes to take part in the same copy/checksum. Split… CONTINUE READING
    7 Citations
    High Performance Reliable File Transfers Using Automatic Many-to-Many Parallelization
    • 4
    • PDF
    A Bloom Filter Based Scalable Data Integrity Check Tool for Large-Scale Dataset
    • Sisi Xiong, F. Wang, Q. Cao
    • Computer Science
    • 2016 1st Joint International Workshop on Parallel Data Storage and data Intensive Scalable Computing Systems (PDSW-DISCS)
    • 2016
    • 5
    Transparent Optimization of Parallel File System I/O via Standard System Tool Enhancement
    • Paul Z. Kolano
    • Computer Science
    • 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum
    • 2013
    • 3
    • PDF
    Best Practices and Lessons Learned from Deploying and Operating Large-Scale Data-Centric Parallel File Systems
    • S. Oral, J. Simmons, +14 authors Arthur S. Bland
    • Computer Science
    • SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
    • 2014
    • 38
    • PDF

    References

    SHOWING 1-10 OF 41 REFERENCES
    PVFS: A Parallel File System for Linux Clusters
    • 1,028
    • PDF
    GPFS: A Shared-Disk File System for Large Computing Clusters
    • 1,426
    • PDF
    Parallel file system testing for the lunatic fringe: the care and feeding of restless I/O power users
    • 36
    • PDF
    A first look at scalable I/O in Linux commands
    • 7
    • PDF
    PARSHA-256- - A New Parallelizable Hash Function and a Multithreaded Implementation
    • 25
    • PDF
    High speed bulk data transfer using the SSH protocol
    • 35
    • PDF
    The Globus Striped GridFTP Framework and Server
    • 644
    • PDF
    OpenMP: an industry standard API for shared-memory programming
    • 2,850
    • PDF
    Lustre: Building a File System for 1,000-node Clusters
    • 382
    • PDF