Using subfiling to improve programming flexibility and performance of parallel shared-file I/O

Kui Gao*, Wei Keng Liao, Arifa Nisar, Alok Choudhary, Robert Ross, Robert Latham

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Scopus citations

Abstract

There are two popular parallel I/O programming styles used by modern scientific computational applications: unique-file and shared-file. Unique-file I/O usually gives satisfactory performance, but its major drawback is that managing a large number of files can overwhelm the task of postsimulation data processing. Shared-file I/O produces fewer files and allows arrays partitioned among processes to be saved in the canonical order. As the number of processors on modern parallel machines increases into thousands and more, the problem size and in turn the global array size also increase proportionally. It is not practical to manage files of size each larger than a few hundreds of GB. Hence, to seek a middle ground between these two I/O styles, we propose a subfiling scheme that divides a large multi-dimensional global array into smaller subarrays, each saved in a smaller file, named subfile. Subfiling is implemented on top of MPI-IO. We also incorporate it into the parallel netCDF library in order to preserve the partitioning information in the netCDF file header, so that the global array can later be reconstructed. In addition, since the subfiling scheme decreases the number of processes sharing a file, it can reduce the overhead of file system's data consistency control. Our experimental results with several I/O benchmarks show that subfiling can provide improved I/O performance.

Original languageEnglish (US)
Title of host publicationICPP-2009 - The 38th International Conference on Parallel Processing
Pages470-477
Number of pages8
DOIs
StatePublished - Dec 1 2009
Event38th International Conference on Parallel Processing, ICPP-2009 - Vienna, Austria
Duration: Sep 22 2009Sep 25 2009

Publication series

NameProceedings of the International Conference on Parallel Processing
ISSN (Print)0190-3918

Other

Other38th International Conference on Parallel Processing, ICPP-2009
CountryAustria
CityVienna
Period9/22/099/25/09

Keywords

  • MPI-IO
  • Parallel netCDF
  • Subfiling

ASJC Scopus subject areas

  • Software
  • Mathematics(all)
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Using subfiling to improve programming flexibility and performance of parallel shared-file I/O'. Together they form a unique fingerprint.

Cite this