Exploiting shared memory to improve parallel I/O performance

Andrew B. Hastings; Alok Choudhary

doi:10.1007/11846802_33

Exploiting shared memory to improve parallel I/O performance

Andrew B. Hastings^*, Alok Choudhary

^*Corresponding author for this work

Electrical and Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

3 Scopus citations

Abstract

We explore several methods utilizing system-wide shared memory to improve the performance of MPI-IO, particularly for noncontiguous file access. We introduce an abstraction called the datatype iterator that permits efficient, dynamic generation of (offset, length) pairs for a given MPI derived datatype. Combining datatype iterators with overlapped I/O and computation, we demonstrate how a shared memory MPI implementation can utilize more than 90% of the available disk bandwidth (in some cases representing a 5× performance improvement over existing methods) even for extreme cases of non-contiguous data-types. We generalize our results to suggest possible parallel I/O performance improvements on systems without global shared memory.

Original language	English (US)
Title of host publication	Recent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings
Publisher	Springer Verlag
Pages	212-221
Number of pages	10
ISBN (Print)	354039110X, 9783540391104
DOIs	https://doi.org/10.1007/11846802_33
State	Published - 2006
Event	13th European PVM/MPI User's Group Meeting - Bonn, Germany Duration: Sep 17 2006 → Sep 20 2006

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	4192 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	13th European PVM/MPI User's Group Meeting
Country	Germany
City	Bonn
Period	9/17/06 → 9/20/06

Keywords

Datatype iterator
MPI-IO
Non-contiguous access
Parallel I/O
Shared memory

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/11846802_33

Fingerprint Dive into the research topics of 'Exploiting shared memory to improve parallel I/O performance'. Together they form a unique fingerprint.

Cite this

Hastings, A. B., & Choudhary, A. (2006). Exploiting shared memory to improve parallel I/O performance. In Recent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings (pp. 212-221). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 4192 LNCS). Springer Verlag. https://doi.org/10.1007/11846802_33

Hastings, Andrew B. ; Choudhary, Alok. / Exploiting shared memory to improve parallel I/O performance. Recent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings. Springer Verlag, 2006. pp. 212-221 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{fe5ab327dc72417d9ba2057f7d7e542e,

title = "Exploiting shared memory to improve parallel I/O performance",

abstract = "We explore several methods utilizing system-wide shared memory to improve the performance of MPI-IO, particularly for noncontiguous file access. We introduce an abstraction called the datatype iterator that permits efficient, dynamic generation of (offset, length) pairs for a given MPI derived datatype. Combining datatype iterators with overlapped I/O and computation, we demonstrate how a shared memory MPI implementation can utilize more than 90% of the available disk bandwidth (in some cases representing a 5× performance improvement over existing methods) even for extreme cases of non-contiguous data-types. We generalize our results to suggest possible parallel I/O performance improvements on systems without global shared memory.",

keywords = "Datatype iterator, MPI-IO, Non-contiguous access, Parallel I/O, Shared memory",

author = "Hastings, {Andrew B.} and Alok Choudhary",

year = "2006",

doi = "10.1007/11846802_33",

language = "English (US)",

isbn = "354039110X",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "212--221",

booktitle = "Recent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings",

}

Hastings, AB & Choudhary, A 2006, Exploiting shared memory to improve parallel I/O performance. in Recent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 4192 LNCS, Springer Verlag, pp. 212-221, 13th European PVM/MPI User's Group Meeting, Bonn, Germany, 9/17/06. https://doi.org/10.1007/11846802_33

Exploiting shared memory to improve parallel I/O performance. / Hastings, Andrew B.; Choudhary, Alok.

Recent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings. Springer Verlag, 2006. p. 212-221 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 4192 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Exploiting shared memory to improve parallel I/O performance

AU - Hastings, Andrew B.

AU - Choudhary, Alok

PY - 2006

Y1 - 2006

N2 - We explore several methods utilizing system-wide shared memory to improve the performance of MPI-IO, particularly for noncontiguous file access. We introduce an abstraction called the datatype iterator that permits efficient, dynamic generation of (offset, length) pairs for a given MPI derived datatype. Combining datatype iterators with overlapped I/O and computation, we demonstrate how a shared memory MPI implementation can utilize more than 90% of the available disk bandwidth (in some cases representing a 5× performance improvement over existing methods) even for extreme cases of non-contiguous data-types. We generalize our results to suggest possible parallel I/O performance improvements on systems without global shared memory.

AB - We explore several methods utilizing system-wide shared memory to improve the performance of MPI-IO, particularly for noncontiguous file access. We introduce an abstraction called the datatype iterator that permits efficient, dynamic generation of (offset, length) pairs for a given MPI derived datatype. Combining datatype iterators with overlapped I/O and computation, we demonstrate how a shared memory MPI implementation can utilize more than 90% of the available disk bandwidth (in some cases representing a 5× performance improvement over existing methods) even for extreme cases of non-contiguous data-types. We generalize our results to suggest possible parallel I/O performance improvements on systems without global shared memory.

KW - Datatype iterator

KW - MPI-IO

KW - Non-contiguous access

KW - Parallel I/O

KW - Shared memory

UR - http://www.scopus.com/inward/record.url?scp=33750263504&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33750263504&partnerID=8YFLogxK

U2 - 10.1007/11846802_33

DO - 10.1007/11846802_33

M3 - Conference contribution

AN - SCOPUS:33750263504

SN - 354039110X

SN - 9783540391104

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 212

EP - 221

BT - Recent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings

PB - Springer Verlag

T2 - 13th European PVM/MPI User's Group Meeting

Y2 - 17 September 2006 through 20 September 2006

ER -

Hastings AB, Choudhary A. Exploiting shared memory to improve parallel I/O performance. In Recent Advances in Parallel Virtual Machine and Message Passing Interface - 13th European PVM/MPI User's Group Meeting, Proceedings. Springer Verlag. 2006. p. 212-221. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/11846802_33

Exploiting shared memory to improve parallel I/O performance

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Cite this