DTF: An I/O arbitration framework for multi-component data processing workflows

Tatiana V. Martsinkevich*, Balazs Gerofi, Guo Yuan Lien, Seiya Nishizawa, Wei keng Liao, Takemasa Miyoshi, Hirofumi Tomita, Yutaka Ishikawa, Alok Choudhary

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Multi-component workflows, where one component performs a particular transformation with the data and passes it on to the next component, is a common way of performing complex computations. Using components as building blocks we can apply sophisticated data processing algorithms to large volumes of data. Because the components may be developed independently, they often use file I/O and the Parallel File System to pass data. However, as the data volume increases, file I/O quickly becomes the bottleneck in such workflows. In this work, we propose an I/O arbitration framework called DTF to alleviate this problem by silently replacing file I/O with direct data transfer between the components. DTF treats file I/O calls as I/O requests and performs I/O request matching to perform data movement. Currently, the framework works with PnetCDF-based multi-component workflows. It requires minimal modifications to applications and allows the user to easily control I/O flow via the framework’s configuration file.

Original languageEnglish (US)
Title of host publicationHigh Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings
EditorsMichele Weiland, David Keyes, Carsten Trinitis, Rio Yokota
PublisherSpringer Verlag
Pages63-80
Number of pages18
ISBN (Print)9783319920399
DOIs
StatePublished - 2018
Event33rd International Conference on ISC High Performance, 2018 - Frankfurt, Germany
Duration: Jun 24 2018Jun 28 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10876 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other33rd International Conference on ISC High Performance, 2018
Country/TerritoryGermany
CityFrankfurt
Period6/24/186/28/18

Keywords

  • I/O arbitration
  • I/O performance
  • Multi-component workflow
  • Workflow coupling

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'DTF: An I/O arbitration framework for multi-component data processing workflows'. Together they form a unique fingerprint.

Cite this