Abstract
This paper presents the design and evaluation of a multi-threaded runtime library for parallel I/O. We extend the multi-threading concept to separate the compute and I/O tasks in two separate threads of control. Multi-threading in our design permits a) asynchronous I/O even if the underlying file system does not support asynchronous I/O; b) copy avoidance from the I/O thread to the compute thread by sharing address space; and c) a capability to perform collective I/O asynchronously without blocking the compute threads. Further, this paper presents techniques for collective I/O which maximize load balance and concurrency while reducing communication overhead in an integrated fashion. Performance results on IBM SP2 for various data distributions and access patterns are presented. The results show that there is a tradeoff between the amount of concurrency in I/O and the buffer size designated for I/O; and there is an optimal buffer size beyond which benefits of larger requests diminish due to large communication overheads.
| Original language | English (US) |
|---|---|
| Pages (from-to) | 368-373 |
| Number of pages | 6 |
| Journal | Proceedings of the International Parallel Processing Symposium, IPPS |
| State | Published - 1997 |
| Event | Proceedings of the 1997 11th International Parallel Processing Symposium, IPPS 97 - Geneva, Switz Duration: Apr 1 1997 → Apr 5 1997 |
ASJC Scopus subject areas
- Hardware and Architecture