Parallel netCDF: A high-performance scientific I/O interface

Jianwei Li, Wei Keng Liao, Alok Choudhary, Robert Ross, Rajeev Thakur, William Gropp, Rob Latham, Andrew Siegel, Brad Gallagher, Michael Zingale

Research output: Chapter in Book/Report/Conference proceedingConference contribution

167 Scopus citations

Abstract

Dataset storage, exchange, and access play a critical role in scientific applications. For such purposes netCDF serves as a portable, efficient file format and programming interface, which is popular in numerous scientific application domains. However, the original interface does not provide an efficient mechanism for parallel data storage and access. In this work, we present a new parallel interface for writing and reading netCDF datasets. This interface is derived with minimal changes from the serial netCDF interface but defines semantics for parallel access and is tailored for high performance. The underlying parallel I/O is achieved through MPI-IO, allowing for substantial performance gains through the use of collective I/O optimizations. We compare the implementation strategies and performance with HDF5. Our tests indicate programming convenience and significant I/O performance improvement with this parallel netCDF (PnetCDF) interface.

Original languageEnglish (US)
Title of host publicationProceedings of the 2003 ACM/IEEE Conference on Supercomputing, SC 2003
DOIs
StatePublished - Dec 1 2003
Event2003 ACM/IEEE Conference on Supercomputing, SC 2003 - Phoenix, AZ, United States
Duration: Nov 15 2003Nov 21 2003

Publication series

NameProceedings of the 2003 ACM/IEEE Conference on Supercomputing, SC 2003

Other

Other2003 ACM/IEEE Conference on Supercomputing, SC 2003
CountryUnited States
CityPhoenix, AZ
Period11/15/0311/21/03

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Parallel netCDF: A high-performance scientific I/O interface'. Together they form a unique fingerprint.

  • Cite this

    Li, J., Liao, W. K., Choudhary, A., Ross, R., Thakur, R., Gropp, W., Latham, R., Siegel, A., Gallagher, B., & Zingale, M. (2003). Parallel netCDF: A high-performance scientific I/O interface. In Proceedings of the 2003 ACM/IEEE Conference on Supercomputing, SC 2003 (Proceedings of the 2003 ACM/IEEE Conference on Supercomputing, SC 2003). https://doi.org/10.1145/1048935.1050189