Integration of burst buffer in high-level parallel I/O library for exa-scale computing era

Kaiyuan Hou, Reda Al-Bahrani, Esteban Rangel, Ankit Agrawal, Robert Latham, Robert Ross, Alok Choudhary, Wei Keng Liao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

While the computing power of supercomputers continues to improve at an astonishing rate, companion I/O systems are struggling to keep up in performance. To mitigate the performance gap, several supercomputing systems have been configured to incorporate burst buffers into their I/O stack; the exact role of which, however, still remains unclear. In this paper, we examine the features of burst buffers and study their impact on application I/O performance. Our goal is to demonstrate that burst buffers can be utilized by parallel I/O libraries to significantly improve performance. To this end, we developed an I/O driver in PnetCDF that uses a log-based format to store individual I/O requests on the burst buffer-later to be flushed to the parallel file system as one request. We evaluated our implementation by running standard I/O benchmarks on Cori, a Cray XC40 supercomputer at NERSC with a centralized burst buffer system, and Theta, a Cray XC40 supercomputer at ALCF with locally available SSDs. Our results show that IO aggregation is a promising role for burst buffers in high-level I/O libraries.

Original languageEnglish (US)
Title of host publicationProceedings of PDSW-DISCS 2018
Subtitle of host publication3rd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1-12
Number of pages12
ISBN (Electronic)9781728101927
DOIs
StatePublished - Feb 8 2019
Event3rd IEEE/ACM Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2018 - Dallas, United States
Duration: Nov 12 2018 → …

Publication series

NameProceedings of PDSW-DISCS 2018: 3rd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference3rd IEEE/ACM Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2018
CountryUnited States
CityDallas
Period11/12/18 → …

Keywords

  • Exa-scale
  • Log
  • burst-buffer

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Hardware and Architecture
  • Software

Fingerprint Dive into the research topics of 'Integration of burst buffer in high-level parallel I/O library for exa-scale computing era'. Together they form a unique fingerprint.

Cite this