Native data representation: An efficient wire format for high-performance distributed computing

Greg Eisenhauer, Fabián E. Bustamante, Karsten Schwan

Research output: Contribution to journalArticlepeer-review

16 Scopus citations

Abstract

New trends in high-performance software development such as tool- and component-based approaches have increased the need for flexible and high-performance communication systems. When trying to reap the well-known benefits of these approaches, the question of what communication infrastructure should be used to link the various components arises. In this context, flexibility and high-performance seem to be incompatible goals. Traditional HPC-style communication libraries, such as MPI, offer good performance, but are not intended for loosely-coupled systems. Object- and metadata-based approaches like XML offer the needed plug-and-play flexibility, but with significantly lower performance. We observe that the flexibility and baseline performance of data exchange systems are strongly determined by their wire formats, or by how they represent data for transmission in heterogeneous environments. After examining the performance implications of using a number of different wire formats, we propose an alternative approach for flexible high-performance data exchange, Native Data Representation, and evaluate its current implementation in the Portable Binary I/O library.

Original languageEnglish (US)
Pages (from-to)1234-1246
Number of pages13
JournalIEEE Transactions on Parallel and Distributed Systems
Volume13
Issue number12
DOIs
StatePublished - Dec 2002

Keywords

  • Communication
  • Distributed computing
  • High-performance
  • Wire format

ASJC Scopus subject areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'Native data representation: An efficient wire format for high-performance distributed computing'. Together they form a unique fingerprint.

Cite this