DTF: An I/O arbitration framework for multi-component data processing workflows

Tatiana V. Martsinkevich; Balazs Gerofi; Guo Yuan Lien; Seiya Nishizawa; Wei keng Liao; Takemasa Miyoshi; Hirofumi Tomita; Yutaka Ishikawa; Alok Choudhary

doi:10.1007/978-3-319-92040-5_4

DTF: An I/O arbitration framework for multi-component data processing workflows

Tatiana V. Martsinkevich^*, Balazs Gerofi, Guo Yuan Lien, Seiya Nishizawa, Wei keng Liao, Takemasa Miyoshi, Hirofumi Tomita, Yutaka Ishikawa, Alok Choudhary

^*Corresponding author for this work

Electrical and Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Scopus citations

Abstract

Multi-component workflows, where one component performs a particular transformation with the data and passes it on to the next component, is a common way of performing complex computations. Using components as building blocks we can apply sophisticated data processing algorithms to large volumes of data. Because the components may be developed independently, they often use file I/O and the Parallel File System to pass data. However, as the data volume increases, file I/O quickly becomes the bottleneck in such workflows. In this work, we propose an I/O arbitration framework called DTF to alleviate this problem by silently replacing file I/O with direct data transfer between the components. DTF treats file I/O calls as I/O requests and performs I/O request matching to perform data movement. Currently, the framework works with PnetCDF-based multi-component workflows. It requires minimal modifications to applications and allows the user to easily control I/O flow via the framework’s configuration file.

Original language	English (US)
Title of host publication	High Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings
Editors	Michele Weiland, David Keyes, Carsten Trinitis, Rio Yokota
Publisher	Springer Verlag
Pages	63-80
Number of pages	18
ISBN (Print)	9783319920399
DOIs	https://doi.org/10.1007/978-3-319-92040-5_4
State	Published - 2018
Event	33rd International Conference on ISC High Performance, 2018 - Frankfurt, Germany Duration: Jun 24 2018 → Jun 28 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10876 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	33rd International Conference on ISC High Performance, 2018
Country/Territory	Germany
City	Frankfurt
Period	6/24/18 → 6/28/18

Keywords

I/O arbitration
I/O performance
Multi-component workflow
Workflow coupling

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-319-92040-5_4

Cite this

Martsinkevich, T. V., Gerofi, B., Lien, G. Y., Nishizawa, S., Liao, W. K., Miyoshi, T., Tomita, H., Ishikawa, Y., & Choudhary, A. (2018). DTF: An I/O arbitration framework for multi-component data processing workflows. In M. Weiland, D. Keyes, C. Trinitis, & R. Yokota (Eds.), High Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings (pp. 63-80). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10876 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-92040-5_4

Martsinkevich, Tatiana V. ; Gerofi, Balazs ; Lien, Guo Yuan et al. / DTF : An I/O arbitration framework for multi-component data processing workflows. High Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings. editor / Michele Weiland ; David Keyes ; Carsten Trinitis ; Rio Yokota. Springer Verlag, 2018. pp. 63-80 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{3d3820b20bfc4e38a30a81ec39838b40,

title = "DTF: An I/O arbitration framework for multi-component data processing workflows",

abstract = "Multi-component workflows, where one component performs a particular transformation with the data and passes it on to the next component, is a common way of performing complex computations. Using components as building blocks we can apply sophisticated data processing algorithms to large volumes of data. Because the components may be developed independently, they often use file I/O and the Parallel File System to pass data. However, as the data volume increases, file I/O quickly becomes the bottleneck in such workflows. In this work, we propose an I/O arbitration framework called DTF to alleviate this problem by silently replacing file I/O with direct data transfer between the components. DTF treats file I/O calls as I/O requests and performs I/O request matching to perform data movement. Currently, the framework works with PnetCDF-based multi-component workflows. It requires minimal modifications to applications and allows the user to easily control I/O flow via the framework{\textquoteright}s configuration file.",

keywords = "I/O arbitration, I/O performance, Multi-component workflow, Workflow coupling",

author = "Martsinkevich, {Tatiana V.} and Balazs Gerofi and Lien, {Guo Yuan} and Seiya Nishizawa and Liao, {Wei keng} and Takemasa Miyoshi and Hirofumi Tomita and Yutaka Ishikawa and Alok Choudhary",

note = "Publisher Copyright: {\textcopyright} 2018, Springer International Publishing AG, part of Springer Nature.; 33rd International Conference on ISC High Performance, 2018 ; Conference date: 24-06-2018 Through 28-06-2018",

year = "2018",

doi = "10.1007/978-3-319-92040-5_4",

language = "English (US)",

isbn = "9783319920399",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "63--80",

editor = "Michele Weiland and David Keyes and Carsten Trinitis and Rio Yokota",

booktitle = "High Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings",

}

Martsinkevich, TV, Gerofi, B, Lien, GY, Nishizawa, S, Liao, WK, Miyoshi, T, Tomita, H, Ishikawa, Y & Choudhary, A 2018, DTF: An I/O arbitration framework for multi-component data processing workflows. in M Weiland, D Keyes, C Trinitis & R Yokota (eds), High Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10876 LNCS, Springer Verlag, pp. 63-80, 33rd International Conference on ISC High Performance, 2018, Frankfurt, Germany, 6/24/18. https://doi.org/10.1007/978-3-319-92040-5_4

DTF: An I/O arbitration framework for multi-component data processing workflows. / Martsinkevich, Tatiana V.; Gerofi, Balazs; Lien, Guo Yuan et al.
High Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings. ed. / Michele Weiland; David Keyes; Carsten Trinitis; Rio Yokota. Springer Verlag, 2018. p. 63-80 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10876 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - DTF

T2 - 33rd International Conference on ISC High Performance, 2018

AU - Martsinkevich, Tatiana V.

AU - Gerofi, Balazs

AU - Lien, Guo Yuan

AU - Nishizawa, Seiya

AU - Liao, Wei keng

AU - Miyoshi, Takemasa

AU - Tomita, Hirofumi

AU - Ishikawa, Yutaka

AU - Choudhary, Alok

PY - 2018

Y1 - 2018

N2 - Multi-component workflows, where one component performs a particular transformation with the data and passes it on to the next component, is a common way of performing complex computations. Using components as building blocks we can apply sophisticated data processing algorithms to large volumes of data. Because the components may be developed independently, they often use file I/O and the Parallel File System to pass data. However, as the data volume increases, file I/O quickly becomes the bottleneck in such workflows. In this work, we propose an I/O arbitration framework called DTF to alleviate this problem by silently replacing file I/O with direct data transfer between the components. DTF treats file I/O calls as I/O requests and performs I/O request matching to perform data movement. Currently, the framework works with PnetCDF-based multi-component workflows. It requires minimal modifications to applications and allows the user to easily control I/O flow via the framework’s configuration file.

AB - Multi-component workflows, where one component performs a particular transformation with the data and passes it on to the next component, is a common way of performing complex computations. Using components as building blocks we can apply sophisticated data processing algorithms to large volumes of data. Because the components may be developed independently, they often use file I/O and the Parallel File System to pass data. However, as the data volume increases, file I/O quickly becomes the bottleneck in such workflows. In this work, we propose an I/O arbitration framework called DTF to alleviate this problem by silently replacing file I/O with direct data transfer between the components. DTF treats file I/O calls as I/O requests and performs I/O request matching to perform data movement. Currently, the framework works with PnetCDF-based multi-component workflows. It requires minimal modifications to applications and allows the user to easily control I/O flow via the framework’s configuration file.

KW - I/O arbitration

KW - I/O performance

KW - Multi-component workflow

KW - Workflow coupling

UR - http://www.scopus.com/inward/record.url?scp=85048544862&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048544862&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-92040-5_4

DO - 10.1007/978-3-319-92040-5_4

M3 - Conference contribution

AN - SCOPUS:85048544862

SN - 9783319920399

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 63

EP - 80

BT - High Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings

A2 - Weiland, Michele

A2 - Keyes, David

A2 - Trinitis, Carsten

A2 - Yokota, Rio

PB - Springer Verlag

Y2 - 24 June 2018 through 28 June 2018

ER -

Martsinkevich TV, Gerofi B, Lien GY, Nishizawa S, Liao WK, Miyoshi T et al. DTF: An I/O arbitration framework for multi-component data processing workflows. In Weiland M, Keyes D, Trinitis C, Yokota R, editors, High Performance Computing - 33rd International Conference, ISC High Performance 2018, Proceedings. Springer Verlag. 2018. p. 63-80. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-92040-5_4

DTF: An I/O arbitration framework for multi-component data processing workflows

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this