Evaluating I/O characteristics and methods for storing structured scientific data

Avery Ching*, Alok Choudhary, Wei Keng Liao, Lee Ward, Neil Pundit

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

43 Scopus citations

Abstract

Many large-scale scientific simulations generate large, structured multi-dimensional datasets. Data is stored at various intervals on high performance I/O storage systems for checkpointing, post-processing, and visualization. Data storage is very I/O intensive and can dominate the overall running time of an application, depending on the characteristics of the I/O access pattern. Our NCIO benchmark determines how I/O characteristics greatly affect performance (up to 2 orders of magnitude) and provides scientific application developers with guidelines for improvement. In this paper, we examine the impact of various I/O parameters and methods when using the MPI-IO interface to store structured scientific data in an optimized parallel file system.

Original languageEnglish (US)
Title of host publication20th International Parallel and Distributed Processing Symposium, IPDPS 2006
PublisherIEEE Computer Society
ISBN (Print)1424400546, 9781424400546
DOIs
StatePublished - Jan 1 2006
Event20th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2006 - Rhodes Island, Greece
Duration: Apr 25 2006Apr 29 2006

Publication series

Name20th International Parallel and Distributed Processing Symposium, IPDPS 2006
Volume2006

Other

Other20th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2006
CountryGreece
CityRhodes Island
Period4/25/064/29/06

ASJC Scopus subject areas

  • Engineering(all)

Fingerprint Dive into the research topics of 'Evaluating I/O characteristics and methods for storing structured scientific data'. Together they form a unique fingerprint.

  • Cite this

    Ching, A., Choudhary, A., Liao, W. K., Ward, L., & Pundit, N. (2006). Evaluating I/O characteristics and methods for storing structured scientific data. In 20th International Parallel and Distributed Processing Symposium, IPDPS 2006 [1639306] (20th International Parallel and Distributed Processing Symposium, IPDPS 2006; Vol. 2006). IEEE Computer Society. https://doi.org/10.1109/IPDPS.2006.1639306