Exploiting inter-file access patterns using multi-collective I/O

Gokhan Memik, Mahmut Kandemir, Alok Choudhary

Research output: Contribution to conferencePaper

7 Scopus citations

Abstract

This paper introduces a new concept called Multi-Collective I/O (MCIO) that extends conventional collective I/O to optimize I/O accesses to multiple arrays simultaneously. In this approach, as in collective I/O, multiple processors co-ordinate to perform I/O on behalf of each other if doing so improves overall I/O time. However, unlike collective I/O, MCIO considers multiple arrays simultaneously; that is, it has a more global view of the overall I/O behavior exhibited by application. This paper shows that determining optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem (also called the assignment problem). Both of the heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our preliminary results show that MCIO out-performs collective I/O by as much as . Our runtime library-based implementation can be used by users as well as optimizing compilers. Based on our results, we recommend future library designers for I/O-intensive applications to include MCIO in their suite of optimizations.

Original languageEnglish (US)
StatePublished - Jan 1 2002
Event2002 USENIX Conference on File and Storage Technologies, FAST 2002 - Monterey, United States
Duration: Jan 28 2002Jan 30 2002

Conference

Conference2002 USENIX Conference on File and Storage Technologies, FAST 2002
CountryUnited States
CityMonterey
Period1/28/021/30/02

ASJC Scopus subject areas

  • Software
  • Computer Networks and Communications
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Exploiting inter-file access patterns using multi-collective I/O'. Together they form a unique fingerprint.

  • Cite this

    Memik, G., Kandemir, M., & Choudhary, A. (2002). Exploiting inter-file access patterns using multi-collective I/O. Paper presented at 2002 USENIX Conference on File and Storage Technologies, FAST 2002, Monterey, United States.