A scalable message passing interface implementation of an Ad-Hoc parallel I/o system

Florin Isaila*, Francisco Javier Garcia Blas, Jesús Carretero, Wei Keng Liao, Alok Choudhary

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

10 Scopus citations

Abstract

In this paper we present the novel design, implementation, and evaluation of an ad-hoc parallel I/O system (AHPIOS). AHPIOS is the first scalable parallel I/O system completely implemented in the Message Passing Interface (MPI). The MPI implementation brings the advantages of portability, scalability and high performance. AHPIOS allows MPI applications to dynamically manage and scale distributed partitions in a convenient way. The configuration of both the MPI-IO and the storage management system is unified and allows for a tight integration of the optimizations of these layers. AHPIOS partitions are elastic: they conveniently scale up and down with the number of resources. We develop two collective I/O strategies, which leverage a two-tiered cooperative cache in order to exploit the spatial locality of data-intensive parallel applications. The file access latency is hidden from the applications through an asynchronous data staging strategy. The two-tiered cooperative cache scales with both the number of processors and storage resources. Our experimental section demonstrates that, with various optimizations, integrated AHPIOS offers a substantial performance benefit over the traditional MPI-IO solutions on both PVFS or Lustre parallel file systems.

Original languageEnglish (US)
Pages (from-to)164-184
Number of pages21
JournalInternational Journal of High Performance Computing Applications
Volume24
Issue number2
DOIs
StatePublished - May 2010

Keywords

  • Distributed file systems
  • Parallel I/O
  • Parallel systems
  • Parallelism and concurrency

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'A scalable message passing interface implementation of an Ad-Hoc parallel I/o system'. Together they form a unique fingerprint.

Cite this