Cooperative write-behind data buffering for MPI I/O

Wei Keng Liao*, Kenin Coloma, Alok Choudhary, Lee Ward

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

12 Scopus citations

Abstract

Many large-scale production parallel programs often run for a very long time and require data checkpoint periodically to save the state of the computation for program restart and/or tracing the progress. Such a write-only pattern has become a dominant part of an application's I/O workload and implies the importance of its optimization. Existing approaches for write-behind data buffering at both file system and MPI I/O levels have been proposed, but challenges still exist for efficient design to maintain data consistency among distributed buffers. To address this problem, we propose a buffering scheme that coordinates the compute processes to achieve the consistency control. Different from other earlier work, our design can be applied to files opened in read-write mode and handle the patterns with mixed MPI collective and independent I/O calls. Performance evaluation using BTIO and FLASH IO benchmarks is presented, which shows a significant improvement over the method without buffering.

Original languageEnglish (US)
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages102-109
Number of pages8
DOIs
StatePublished - Dec 1 2005
Event12th European PVM/MPI Users' Group Meeting - Recent Advances in Parallel Virtual Machine and Message Passing Interface - Sorrento, Italy
Duration: Sep 18 2005Sep 21 2005

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume3666 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other12th European PVM/MPI Users' Group Meeting - Recent Advances in Parallel Virtual Machine and Message Passing Interface
Country/TerritoryItaly
CitySorrento
Period9/18/059/21/05

Keywords

  • Data buffering
  • File consistency
  • I/O thread
  • MPI I/O
  • Write behind

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Cooperative write-behind data buffering for MPI I/O'. Together they form a unique fingerprint.

Cite this