Native Data Representation: An EÆcient Wire Format for High Performance Computing GIT-CC-01-18

Abstract

Flexible and high-performance data exchange is becoming increasingly important. This trend is due in part to the growing interest among highperformance researchers in tooland component-based approaches to software development. In trying to reap the well-known bene ts of these approaches, the question of what communications infrastructure should be used to link the various application components arises. Traditional HPC-style communication libraries such as MPI o er good performance, but are not intended for loosely-coupled systems. Objectand metadatabased approaches like XML o er the needed plug-and-play exibility, but with signi cantly lower performance. We observe that the exibility and baseline performance of data exchange systems are strongly determined by their `wire formats', or by how they represent data for transmission in the heterogeneous environments. Upon examining the performance implications of using a number of di erent wire formats, we propose an alternative approach to exible high-performance data exchange, Native Data Representation, and evaluate its current implementation in the Portable Binary I/O library.

13 Figures and Tables

Cite this paper

@inproceedings{EisenhauerNativeDR, title={Native Data Representation: An E{\aE}cient Wire Format for High Performance Computing GIT-CC-01-18}, author={Greg Eisenhauer and Karsten Schwan} }