Meta-data management system for high-performance large-scale scientific data access

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Scopus citations

Abstract

Many scientific applications manipulate large amount of data and, therefore, are parallelized on high-performance computing systems to take advantage of their computational power and memory space. The size of data processed by these large-scale applications can easily overwhelm the disk capacity of most systems. Thus, tertiary storage devices are used to store the data. The parallelization of this type of applications requires understanding of not only the data partition pattern among multiple processors but also the underlying storage architectures and the data storage pattern. In this paper, we present a meta-data management system which uses a database to record the information of datasets and manage these meta data to provide suitable I/O interface. As a result, users specify dataset names instead of data physical location to access data using optimal I/O calls without knowing the underlying storage structure. We use an astrophysics application to demonstrate that the management system can provide convenient programming environment with negligible database access overhead.

Original languageEnglish (US)
Title of host publicationHigh Performance Computing - HiPC 2000 - 7th International Conference, Proceedings
EditorsMateo Valero, Viktor K. Prasanna, Sriram Vajapeyam
PublisherSpringer Verlag
Pages293-300
Number of pages8
ISBN (Print)3540414290, 9783540414292
DOIs
StatePublished - 2000
Event7th International Conference on High Performance Computing, HiPC 2000 - Bangalore, India
Duration: Dec 17 2000Dec 20 2000

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume1970
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other7th International Conference on High Performance Computing, HiPC 2000
Country/TerritoryIndia
CityBangalore
Period12/17/0012/20/00

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Meta-data management system for high-performance large-scale scientific data access'. Together they form a unique fingerprint.

Cite this