DPFS: A distributed parallel file system

Xiaohui Shen, A. Choudhary

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Scopus citations

Abstract

One of the challenges brought by large-scale scientific applications is how to avoid remote storage access by collectively using enough local storage resources to hold huge amount of data generated by the simulation while providing high performance I/O. DPFS, a Distributed Parallel File System, is designed and implemented to address this problem. DPFS collects locally distributed unused storage resources as a supplement to the internal storage of parallel computing systems to satisfy the storage capacity requirement of large-scale applications. In addition, like parallel file systems, DPFS provides striping mechanisms that divides a file into small pieces and distributes them across multiple storage devices for parallel data access. The unique feature of DPFS is that it provides three file levels with each file level corresponding to a file striping method. In addition to the traditional linear striping method, DPFS also provides a novel multidimensional striping method that can solve performance problems of linear striping for many popular access patterns. Other issues such as load-balancing and user interface are also addressed in DPFS.

Original languageEnglish (US)
Title of host publicationProceedings - International Conference on Parallel Processing, ICPP 2001
EditorsLionel M. Ni, Mateo Valero
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages533-541
Number of pages9
ISBN (Electronic)0769512577
DOIs
StatePublished - 2001
EventInternational Conference on Parallel Processing, ICPP 2001 - Valencia, Spain
Duration: Sep 3 2001Sep 7 2001

Publication series

NameProceedings of the International Conference on Parallel Processing
Volume2001-January
ISSN (Print)0190-3918

Other

OtherInternational Conference on Parallel Processing, ICPP 2001
CountrySpain
CityValencia
Period9/3/019/7/01

Keywords

  • Computational modeling
  • Computer simulation
  • Distributed computing
  • Distributed power generation
  • Ear
  • File systems
  • Large-scale systems
  • Multidimensional systems
  • Parallel processing
  • Spatial databases

ASJC Scopus subject areas

  • Software
  • Mathematics(all)
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'DPFS: A distributed parallel file system'. Together they form a unique fingerprint.

Cite this