A high-performance application data environment for large-scale scientific computations

Xiaohui Shen; Wei-Keng Liao; Alok Nidhi Choudhary; Gokhan Memik; Mahmut Kandemir

doi:10.1109/TPDS.2003.1255638

A high-performance application data environment for large-scale scientific computations

Xiaohui Shen^*, Wei-Keng Liao, Alok Nidhi Choudhary, Gokhan Memik, Mahmut Kandemir

^*Corresponding author for this work

Electrical and Computer Engineering

Research output: Contribution to journal › Article › peer-review

7 Scopus citations

Abstract

Effective high-level data management is becoming an important issue with more and more scientific applications manipulating huge amounts of secondary-storage and tertiary-storage data using parallel processors. A major problem facing the current solutions to this data management problem is that these solutions either require a deep understanding of specific data storage architectures and file layouts to obtain the best performance (as in high-performance storage management systems and parallel file systems), or they sacrifice significant performance in exchange for ease-of-use and portability (as in traditional database management systems). In this paper, we discuss the design, implementation, and evaluation of a novel application development environment for scientific computations. This environment includes a number of components that make it easy for the programmers to code and run their applications without much programming effort and, at the same time, to harness the available computational and storage power on parallel architectures.

Original language	English (US)
Pages (from-to)	1262-1274
Number of pages	13
Journal	IEEE Transactions on Parallel and Distributed Systems
Volume	14
Issue number	12
DOIs	https://doi.org/10.1109/TPDS.2003.1255638
State	Published - Dec 2003

Funding

This research was supported by the Department of Energy under the Accelerated Strategic Computing Initiative (ASCI) Academic Strategic Alliance Program (ASAP) Level 2, under subcontract no. W-7405-ENG-48 from Lawrence Livermore National Laboratories. The authors would like to thank Reagan Moore for discussions and help with SDSC resources. They would also like to thank Mike Wan and Mike Gleicher of SDSC for helping them with the implementation of the volume rendering code and in understanding the SRB and the HPSS. They thank Larry Schoof and Wilbur Johnson for providing the unstructured code used in this paper. They also thank Rick Stevens and Rajeev Thakur of ANL for various discussions on the problem of data management. They also thank Jaechun No for her help with the astrophysics application used in the experiments. Finally, they would like to thank Celeste Matarazzo, John Ambrosiano, and Steve Louis for discussions and their input.

Keywords

Access pattern
Data intensive computing
MDMS
Storage pattern

ASJC Scopus subject areas

Signal Processing
Hardware and Architecture
Computational Theory and Mathematics

Access to Document

10.1109/TPDS.2003.1255638

Cite this

@article{e812ee0808704c93acd7d1f13e0a1c4d,

title = "A high-performance application data environment for large-scale scientific computations",

abstract = "Effective high-level data management is becoming an important issue with more and more scientific applications manipulating huge amounts of secondary-storage and tertiary-storage data using parallel processors. A major problem facing the current solutions to this data management problem is that these solutions either require a deep understanding of specific data storage architectures and file layouts to obtain the best performance (as in high-performance storage management systems and parallel file systems), or they sacrifice significant performance in exchange for ease-of-use and portability (as in traditional database management systems). In this paper, we discuss the design, implementation, and evaluation of a novel application development environment for scientific computations. This environment includes a number of components that make it easy for the programmers to code and run their applications without much programming effort and, at the same time, to harness the available computational and storage power on parallel architectures.",

keywords = "Access pattern, Data intensive computing, MDMS, Storage pattern",

author = "Xiaohui Shen and Wei-Keng Liao and Choudhary, {Alok Nidhi} and Gokhan Memik and Mahmut Kandemir",

note = "Funding Information: This research was supported by the Department of Energy under the Accelerated Strategic Computing Initiative (ASCI) Academic Strategic Alliance Program (ASAP) Level 2, under subcontract no. W-7405-ENG-48 from Lawrence Livermore National Laboratories. The authors would like to thank Reagan Moore for discussions and help with SDSC resources. They would also like to thank Mike Wan and Mike Gleicher of SDSC for helping them with the implementation of the volume rendering code and in understanding the SRB and the HPSS. They thank Larry Schoof and Wilbur Johnson for providing the unstructured code used in this paper. They also thank Rick Stevens and Rajeev Thakur of ANL for various discussions on the problem of data management. They also thank Jaechun No for her help with the astrophysics application used in the experiments. Finally, they would like to thank Celeste Matarazzo, John Ambrosiano, and Steve Louis for discussions and their input.",

year = "2003",

month = dec,

doi = "10.1109/TPDS.2003.1255638",

language = "English (US)",

volume = "14",

pages = "1262--1274",

journal = "IEEE Transactions on Parallel and Distributed Systems",

issn = "1045-9219",

publisher = "IEEE Computer Society",

number = "12",

}

TY - JOUR

T1 - A high-performance application data environment for large-scale scientific computations

AU - Shen, Xiaohui

AU - Liao, Wei-Keng

AU - Choudhary, Alok Nidhi

AU - Memik, Gokhan

AU - Kandemir, Mahmut

N1 - Funding Information: This research was supported by the Department of Energy under the Accelerated Strategic Computing Initiative (ASCI) Academic Strategic Alliance Program (ASAP) Level 2, under subcontract no. W-7405-ENG-48 from Lawrence Livermore National Laboratories. The authors would like to thank Reagan Moore for discussions and help with SDSC resources. They would also like to thank Mike Wan and Mike Gleicher of SDSC for helping them with the implementation of the volume rendering code and in understanding the SRB and the HPSS. They thank Larry Schoof and Wilbur Johnson for providing the unstructured code used in this paper. They also thank Rick Stevens and Rajeev Thakur of ANL for various discussions on the problem of data management. They also thank Jaechun No for her help with the astrophysics application used in the experiments. Finally, they would like to thank Celeste Matarazzo, John Ambrosiano, and Steve Louis for discussions and their input.

PY - 2003/12

Y1 - 2003/12

N2 - Effective high-level data management is becoming an important issue with more and more scientific applications manipulating huge amounts of secondary-storage and tertiary-storage data using parallel processors. A major problem facing the current solutions to this data management problem is that these solutions either require a deep understanding of specific data storage architectures and file layouts to obtain the best performance (as in high-performance storage management systems and parallel file systems), or they sacrifice significant performance in exchange for ease-of-use and portability (as in traditional database management systems). In this paper, we discuss the design, implementation, and evaluation of a novel application development environment for scientific computations. This environment includes a number of components that make it easy for the programmers to code and run their applications without much programming effort and, at the same time, to harness the available computational and storage power on parallel architectures.

AB - Effective high-level data management is becoming an important issue with more and more scientific applications manipulating huge amounts of secondary-storage and tertiary-storage data using parallel processors. A major problem facing the current solutions to this data management problem is that these solutions either require a deep understanding of specific data storage architectures and file layouts to obtain the best performance (as in high-performance storage management systems and parallel file systems), or they sacrifice significant performance in exchange for ease-of-use and portability (as in traditional database management systems). In this paper, we discuss the design, implementation, and evaluation of a novel application development environment for scientific computations. This environment includes a number of components that make it easy for the programmers to code and run their applications without much programming effort and, at the same time, to harness the available computational and storage power on parallel architectures.

KW - Access pattern

KW - Data intensive computing

KW - MDMS

KW - Storage pattern

UR - http://www.scopus.com/inward/record.url?scp=0742268947&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0742268947&partnerID=8YFLogxK

U2 - 10.1109/TPDS.2003.1255638

DO - 10.1109/TPDS.2003.1255638

M3 - Article

AN - SCOPUS:0742268947

SN - 1045-9219

VL - 14

SP - 1262

EP - 1274

JO - IEEE Transactions on Parallel and Distributed Systems

JF - IEEE Transactions on Parallel and Distributed Systems

IS - 12

ER -

A high-performance application data environment for large-scale scientific computations

Abstract

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this