MTIO: A multi-threaded parallel I/O system

Sachin More*, Alok Nidhi Choudhary, Ian Foster, Ming Q. Xu

*Corresponding author for this work

Research output: Contribution to journalConference article

18 Scopus citations

Abstract

This paper presents the design and evaluation of a multi-threaded runtime library for parallel I/O. We extend the multi-threading concept to separate the compute and I/O tasks in two separate threads of control. Multi-threading in our design permits a) asynchronous I/O even if the underlying file system does not support asynchronous I/O; b) copy avoidance from the I/O thread to the compute thread by sharing address space; and c) a capability to perform collective I/O asynchronously without blocking the compute threads. Further, this paper presents techniques for collective I/O which maximize load balance and concurrency while reducing communication overhead in an integrated fashion. Performance results on IBM SP2 for various data distributions and access patterns are presented. The results show that there is a tradeoff between the amount of concurrency in I/O and the buffer size designated for I/O; and there is an optimal buffer size beyond which benefits of larger requests diminish due to large communication overheads.

Original languageEnglish (US)
Pages (from-to)368-373
Number of pages6
JournalProceedings of the International Parallel Processing Symposium, IPPS
StatePublished - Jan 1 1997
EventProceedings of the 1997 11th International Parallel Processing Symposium, IPPS 97 - Geneva, Switz
Duration: Apr 1 1997Apr 5 1997

ASJC Scopus subject areas

  • Hardware and Architecture

Fingerprint Dive into the research topics of 'MTIO: A multi-threaded parallel I/O system'. Together they form a unique fingerprint.

  • Cite this