Experiences in using os-level virtualization for block I/O

  title={Experiences in using os-level virtualization for block I/O},
  author={Dan Huang and Jun Wang and Qing Liu and Jiangling Yin and Xuhong Zhang and Xunchao Chen},
  booktitle={PDSW '15},
Today, HPC clusters commonly use Resource Management Systems such as PBS and TORQUE to share physical resources. These systems enable resources to be shared by assigning nodes to users exclusively in non-overlapping time slots. With virtualization technology, users can run their applications on the same node with low mutual interference. However, the overhead introduced by the virtual machine monitor or hypervisor is too high to be accepted, because efficiency is key to many HPC applications… Expand
SideIO: A Side I/O system framework for hybrid scientific workflow
It is found that SideIO is able to achieve comparable read/write I/O performance in small-medium sized HPC clusters equipped with PFS and achieves up to 3x speedups for hybrid scientific workflow applications compared with current solutions. Expand
Achieving Load Balance for Parallel Data Access on Distributed File Systems
Novel methods, referred to as Opass, are proposed to optimize parallel data reads, as well as to reduce the imbalance of parallel writes on distributed file systems to benefit parallel data-intensive analysis and balanced data access. Expand


MultiLanes: providing virtualized storage for OS-level virtualization on many cores
The evaluation of the prototype system built for Linux container (LXC) on a 32-core machine with both a RAM disk and a modern flash-based SSD demonstrates that MultiLanes scales much better than Linux in micro- and macro-benchmarks, bringing significant performance improvements, and thatMultiLanes with SFS can further reduce the contention within each single container. Expand
Performance Evaluation of Container-based Virtualization for High Performance Computing Environments
The features of three container-based operating systems virtualization tools (LXC, Docker and Singularity) are presented and the disk I/O performance, Memory, Network bandwidth and GPU performance are tested for the COS technologies vs bare metal. Expand
Container-based operating system virtualization: a scalable, high-performance alternative to hypervisors
This paper describes the design and implementation of Linux-VServer, an alternative to hypervisors that is better suited to scenarios that require system virtualization with high degrees of both isolation and efficiency, and shows how Linux- VServer provides comparable support for isolation and superior system efficiency. Expand
Evaluating and Modeling Virtualization Performance Overhead for Cloud Environments
This paper presents experimental results on two popular state-of-the-art virtualization platforms, Citrix XenServer 5.5 and VMware ESX 4.0, and proposes a basic, generic performance prediction model for the two different types of hypervisor architectures. Expand
Performance Evaluation of Container-Based Virtualization for High Performance Computing Environments
This work conducted a number of experiments in order to perform an in-depth performance evaluation of container-based virtualization for HPC, and compared them with Xen, which is a representative of the traditional hypervisor-basedvirtualization systems used today. Expand
Locality-aware dynamic VM reconfiguration on MapReduce clouds
This paper proposes a dynamic VM reconfiguration technique for data-intensive computing on clouds, called Dynamic Resource Reconfiguration (DRR), which can adjust the computing capability of individual VMs to maximize the utilization of resources. Expand
Cost-effective cloud HPC resource provisioning by building Semi-Elastic virtual clusters
This paper proposes a Semi-Elastic Cluster (SEC) computing model for organizations to reserve and dynamically resize a virtual cloud-based cluster, as well as an online reserved instance provisioning algorithm based on job history, and presents a set of integrated batch scheduling plus resource scaling strategies uniquely enabled by SEC. Expand
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center
The results show that Mesos can achieve near-optimal data locality when sharing the cluster among diverse frameworks, can scale to 50,000 (emulated) nodes, and is resilient to failures. Expand
ACIC: Automatic cloud I/O configurator for HPC applications
  • Mingliang Liu, Ye Jin, +4 authors Wenguang Chen
  • Computer Science
  • 2013 SC - International Conference for High Performance Computing, Networking, Storage and Analysis (SC)
  • 2013
This paper proposes ACIC, which takes a given application running on a given cloud platform, and automatically searches for optimized I/O system configurations, and utilizes machine learning models to perform black-box performance/cost predictions. Expand
MapReduce: Simplified Data Processing on Large Clusters
This paper presents the implementation of MapReduce, a programming model and an associated implementation for processing and generating large data sets that runs on a large cluster of commodity machines and is highly scalable. Expand