Multicollective I/O: A technique for exploiting inter-file access patterns

Gokhan Memik*, Mahmut T. Kandemir, Wei-Keng Liao, Alok Nidhi Choudhary

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Scopus citations


The increasing gap between processor cycle times and access times to storage devices makes it necessary to use powerful optimizations. This is especially true for applications in the parallel computing domain that frequently perform large amounts of file I/O. Collective I/O strategy that coordinates the processes to perform I/O on each other's behalf has demonstrated a significant performance improvement. This article proposes a new concept called Multicollective I/O (MCIO) that expands the collective I/O to allow data from multiple files to be requested in a single I/O request, in contrast to allowing only multiple segments for a single file to be specified together. MCIO considers multiple arrays simultaneously by having a more global view of the overall I/O behavior exhibited by parallel applications. This article shows that determining the optimal MCIO access pattern is an NP-complete problem, and proposes two different heuristics for the access pattern detection problem, also called the assignment problem. Both heuristics have been implemented within a runtime library, and tested using a large-scale scientific application. Our results show that MCIO outperforms collective I/O by as much as 87%. Our runtime library-based implementation can be used by application users as well as by optimizing compilers. Based on our results, we recommend that future library designers for I/O-intensive applications include MCIO in their suite of optimizations.

Original languageEnglish (US)
Pages (from-to)349-369
Number of pages21
JournalACM Transactions on Storage
Issue number3
StatePublished - Aug 2006


  • Collective I/O
  • File accesses
  • Parallel I/O
  • Runtime library
  • Software optimizations

ASJC Scopus subject areas

  • Hardware and Architecture


Dive into the research topics of 'Multicollective I/O: A technique for exploiting inter-file access patterns'. Together they form a unique fingerprint.

Cite this