Exploiting shared memory to improve parallel I/O performance

Andrew B. Hastings*, Alok Choudhary

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

We explore several methods utilizing system-wide shared memory to improve the performance of MPI-IO, particularly for noncontiguous file access. We introduce an abstraction called the datatype iterator that permits efficient, dynamic generation of (offset, length) pairs for a given MPI derived datatype. Combining datatype iterators with overlapped I/O and computation, we demonstrate how a shared memory MPI implementation can utilize more than 90% of the available disk bandwidth (in some cases representing a 5× performance improvement over existing methods) even for extreme cases of non-contiguous data-types. We generalize our results to suggest possible parallel I/O performance improvements on systems without global shared memory.

Original languageEnglish (US)
Title of host publicationRecent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings
PublisherSpringer Verlag
Pages212-221
Number of pages10
ISBN (Print)354039110X, 9783540391104
DOIs
StatePublished - 2006
Event13th European PVM/MPI User's Group Meeting - Bonn, Germany
Duration: Sep 17 2006Sep 20 2006

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume4192 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other13th European PVM/MPI User's Group Meeting
CountryGermany
CityBonn
Period9/17/069/20/06

Keywords

  • Datatype iterator
  • MPI-IO
  • Non-contiguous access
  • Parallel I/O
  • Shared memory

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Exploiting shared memory to improve parallel I/O performance'. Together they form a unique fingerprint.

Cite this