Scalable design and implementations for MPI parallel overlapping I/O

Wei Keng Liao; Kenin Coloma; Alok Choudhary; Lee Ward; Eric Russell; Neil Pundit

doi:10.1109/TPDS.2006.163

Scalable design and implementations for MPI parallel overlapping I/O

Wei Keng Liao^*, Kenin Coloma, Alok Choudhary, Lee Ward, Eric Russell, Neil Pundit

^*Corresponding author for this work

Electrical and Computer Engineering

Research output: Contribution to journal › Article › peer-review

12 Scopus citations

Abstract

We investigate the Message Passing Interface Input/Output (MPI I/O) implementation issues for two overlapping access patterns: the overlaps among processes within a single I/O operation and the overlaps across a sequence of I/O operations. The former case considers whether I/O atomicity can be obtained in the overlapping regions. The latter focuses on the file consistency problem on parallel machines with client-side file caching enabled. Traditional solutions for both overlapping I/O problems use whole file or byte-range file locking to ensure exclusive access to the overlapping regions and bypass the file system cache. Unfortunately, not only can file locking serialize I/O, but it can also increase the aggregate communication overhead between clients and I/O servers. For atomicity, we first differentiate MPI's requirements from the Portable Operating System Interface (POSIX) standard and propose two scalable approaches, graph coloring and process-rank ordering, which can resolve access conflicts and maintain I/O parallelism. For solving the file consistency problem across multiple I/O operations, we propose a method called Persistent File Domains, which tackles cache coherency with additional information and coordination to guarantee safe cache access without using file locks.

Original language	English (US)
Pages (from-to)	1264-1276
Number of pages	13
Journal	IEEE Transactions on Parallel and Distributed Systems
Volume	17
Issue number	11
DOIs	https://doi.org/10.1109/TPDS.2006.163
State	Published - Nov 2006

Funding

This work was supported in part by the US Department of Energy Sandia National Laboratories under subcontract No. PO28264, DOE’s SCiDAC program (Scientific Data Management Center), award No. DE-FC02-01ER25485, and in part by US National Science Foundation NSF grant EIA-0103023. It was also supported in part by NSF cooperative agreement ACI-9619020 through computing resources provided by the National Partnership for Advanced Computational Infrastructure at the San Diego Supercomputer Center. The authors are grateful to Gary Grider and Tyce Mclarty from LLNL, and James Nunez from LANL for their valuable comments to this work. They also acknowledge the use of the SGI Origin 2000 at NCSA.

Keywords

Atomic I/O
Cache coherence
File atomicity
File consistency
MPI
MPI I/O
Overlapping I/O

ASJC Scopus subject areas

Signal Processing
Hardware and Architecture
Computational Theory and Mathematics

Access to Document

10.1109/TPDS.2006.163

Cite this

@article{247680bd4a404f23a7487f1bba3cc618,

title = "Scalable design and implementations for MPI parallel overlapping I/O",

abstract = "We investigate the Message Passing Interface Input/Output (MPI I/O) implementation issues for two overlapping access patterns: the overlaps among processes within a single I/O operation and the overlaps across a sequence of I/O operations. The former case considers whether I/O atomicity can be obtained in the overlapping regions. The latter focuses on the file consistency problem on parallel machines with client-side file caching enabled. Traditional solutions for both overlapping I/O problems use whole file or byte-range file locking to ensure exclusive access to the overlapping regions and bypass the file system cache. Unfortunately, not only can file locking serialize I/O, but it can also increase the aggregate communication overhead between clients and I/O servers. For atomicity, we first differentiate MPI's requirements from the Portable Operating System Interface (POSIX) standard and propose two scalable approaches, graph coloring and process-rank ordering, which can resolve access conflicts and maintain I/O parallelism. For solving the file consistency problem across multiple I/O operations, we propose a method called Persistent File Domains, which tackles cache coherency with additional information and coordination to guarantee safe cache access without using file locks.",

keywords = "Atomic I/O, Cache coherence, File atomicity, File consistency, MPI, MPI I/O, Overlapping I/O",

author = "Liao, {Wei Keng} and Kenin Coloma and Alok Choudhary and Lee Ward and Eric Russell and Neil Pundit",

note = "Funding Information: This work was supported in part by the US Department of Energy Sandia National Laboratories under subcontract No. PO28264, DOE{\textquoteright}s SCiDAC program (Scientific Data Management Center), award No. DE-FC02-01ER25485, and in part by US National Science Foundation NSF grant EIA-0103023. It was also supported in part by NSF cooperative agreement ACI-9619020 through computing resources provided by the National Partnership for Advanced Computational Infrastructure at the San Diego Supercomputer Center. The authors are grateful to Gary Grider and Tyce Mclarty from LLNL, and James Nunez from LANL for their valuable comments to this work. They also acknowledge the use of the SGI Origin 2000 at NCSA.",

year = "2006",

month = nov,

doi = "10.1109/TPDS.2006.163",

language = "English (US)",

volume = "17",

pages = "1264--1276",

journal = "IEEE Transactions on Parallel and Distributed Systems",

issn = "1045-9219",

publisher = "IEEE Computer Society",

number = "11",

}

TY - JOUR

T1 - Scalable design and implementations for MPI parallel overlapping I/O

AU - Liao, Wei Keng

AU - Coloma, Kenin

AU - Choudhary, Alok

AU - Ward, Lee

AU - Russell, Eric

AU - Pundit, Neil

N1 - Funding Information: This work was supported in part by the US Department of Energy Sandia National Laboratories under subcontract No. PO28264, DOE’s SCiDAC program (Scientific Data Management Center), award No. DE-FC02-01ER25485, and in part by US National Science Foundation NSF grant EIA-0103023. It was also supported in part by NSF cooperative agreement ACI-9619020 through computing resources provided by the National Partnership for Advanced Computational Infrastructure at the San Diego Supercomputer Center. The authors are grateful to Gary Grider and Tyce Mclarty from LLNL, and James Nunez from LANL for their valuable comments to this work. They also acknowledge the use of the SGI Origin 2000 at NCSA.

PY - 2006/11

Y1 - 2006/11

N2 - We investigate the Message Passing Interface Input/Output (MPI I/O) implementation issues for two overlapping access patterns: the overlaps among processes within a single I/O operation and the overlaps across a sequence of I/O operations. The former case considers whether I/O atomicity can be obtained in the overlapping regions. The latter focuses on the file consistency problem on parallel machines with client-side file caching enabled. Traditional solutions for both overlapping I/O problems use whole file or byte-range file locking to ensure exclusive access to the overlapping regions and bypass the file system cache. Unfortunately, not only can file locking serialize I/O, but it can also increase the aggregate communication overhead between clients and I/O servers. For atomicity, we first differentiate MPI's requirements from the Portable Operating System Interface (POSIX) standard and propose two scalable approaches, graph coloring and process-rank ordering, which can resolve access conflicts and maintain I/O parallelism. For solving the file consistency problem across multiple I/O operations, we propose a method called Persistent File Domains, which tackles cache coherency with additional information and coordination to guarantee safe cache access without using file locks.

AB - We investigate the Message Passing Interface Input/Output (MPI I/O) implementation issues for two overlapping access patterns: the overlaps among processes within a single I/O operation and the overlaps across a sequence of I/O operations. The former case considers whether I/O atomicity can be obtained in the overlapping regions. The latter focuses on the file consistency problem on parallel machines with client-side file caching enabled. Traditional solutions for both overlapping I/O problems use whole file or byte-range file locking to ensure exclusive access to the overlapping regions and bypass the file system cache. Unfortunately, not only can file locking serialize I/O, but it can also increase the aggregate communication overhead between clients and I/O servers. For atomicity, we first differentiate MPI's requirements from the Portable Operating System Interface (POSIX) standard and propose two scalable approaches, graph coloring and process-rank ordering, which can resolve access conflicts and maintain I/O parallelism. For solving the file consistency problem across multiple I/O operations, we propose a method called Persistent File Domains, which tackles cache coherency with additional information and coordination to guarantee safe cache access without using file locks.

KW - Atomic I/O

KW - Cache coherence

KW - File atomicity

KW - File consistency

KW - MPI

KW - MPI I/O

KW - Overlapping I/O

UR - http://www.scopus.com/inward/record.url?scp=33750100114&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33750100114&partnerID=8YFLogxK

U2 - 10.1109/TPDS.2006.163

DO - 10.1109/TPDS.2006.163

M3 - Article

AN - SCOPUS:33750100114

SN - 1045-9219

VL - 17

SP - 1264

EP - 1276

JO - IEEE Transactions on Parallel and Distributed Systems

JF - IEEE Transactions on Parallel and Distributed Systems

IS - 11

ER -

Scalable design and implementations for MPI parallel overlapping I/O

Abstract

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this