Incremental elasticity for array databases

Jennie Duggan, Michael Stonebraker

Research output: Chapter in Book/Report/Conference proceedingConference contribution

11 Citations (Scopus)

Abstract

Relational databases benefit significantly from elasticity, whereby they execute on a set of changing hardware resources provisioned to match their storage and processing requirements. Such flexibility is especially attractive for scientific databases because their users often have a no-overwrite storage model, in which they delete data only when their available space is exhausted. This results in a database that is regularly growing and expanding its hardware proportionally. Also, scientific databases frequently store their data as multidimensional arrays optimized for spatial querying. This brings about several novel challenges in clustered, skewaware data placement on an elastic shared-nothing database. In this work, we design and implement elasticity for an array database. We address this challenge on two fronts: determining when to expand a database cluster and how to partition the data within it. In both steps we propose incremental approaches, affecting a minimum set of data and nodes, while maintaining high performance. We introduce an algorithm for gradually augmenting an array database's hardware using a closed-loop control system. After the cluster adds nodes, we optimize data placement for n-dimensional arrays. Many of our elastic partitioners incrementally reorganize an array, redistributing data only to new nodes. By combining these two tools, the scientific database efficiently and seamlessly manages its monotonically increasing hardware resources.

Original languageEnglish (US)
Title of host publicationSIGMOD 2014 - Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data
PublisherAssociation for Computing Machinery
Pages409-420
Number of pages12
ISBN (Print)9781450323765
StatePublished - Jan 1 2014
Event2014 ACM SIGMOD International Conference on Management of Data, SIGMOD 2014 - Snowbird, UT, United States
Duration: Jun 22 2014Jun 27 2014

Other

Other2014 ACM SIGMOD International Conference on Management of Data, SIGMOD 2014
CountryUnited States
CitySnowbird, UT
Period6/22/146/27/14

Fingerprint

Elasticity
Hardware
Closed loop control systems
Processing

ASJC Scopus subject areas

  • Software
  • Information Systems

Cite this

Duggan, J., & Stonebraker, M. (2014). Incremental elasticity for array databases. In SIGMOD 2014 - Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data (pp. 409-420). Association for Computing Machinery.
Duggan, Jennie ; Stonebraker, Michael. / Incremental elasticity for array databases. SIGMOD 2014 - Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. Association for Computing Machinery, 2014. pp. 409-420
@inproceedings{603b00c57b7f45889403ae8a7c6921bb,
title = "Incremental elasticity for array databases",
abstract = "Relational databases benefit significantly from elasticity, whereby they execute on a set of changing hardware resources provisioned to match their storage and processing requirements. Such flexibility is especially attractive for scientific databases because their users often have a no-overwrite storage model, in which they delete data only when their available space is exhausted. This results in a database that is regularly growing and expanding its hardware proportionally. Also, scientific databases frequently store their data as multidimensional arrays optimized for spatial querying. This brings about several novel challenges in clustered, skewaware data placement on an elastic shared-nothing database. In this work, we design and implement elasticity for an array database. We address this challenge on two fronts: determining when to expand a database cluster and how to partition the data within it. In both steps we propose incremental approaches, affecting a minimum set of data and nodes, while maintaining high performance. We introduce an algorithm for gradually augmenting an array database's hardware using a closed-loop control system. After the cluster adds nodes, we optimize data placement for n-dimensional arrays. Many of our elastic partitioners incrementally reorganize an array, redistributing data only to new nodes. By combining these two tools, the scientific database efficiently and seamlessly manages its monotonically increasing hardware resources.",
author = "Jennie Duggan and Michael Stonebraker",
year = "2014",
month = "1",
day = "1",
language = "English (US)",
isbn = "9781450323765",
pages = "409--420",
booktitle = "SIGMOD 2014 - Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data",
publisher = "Association for Computing Machinery",

}

Duggan, J & Stonebraker, M 2014, Incremental elasticity for array databases. in SIGMOD 2014 - Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. Association for Computing Machinery, pp. 409-420, 2014 ACM SIGMOD International Conference on Management of Data, SIGMOD 2014, Snowbird, UT, United States, 6/22/14.

Incremental elasticity for array databases. / Duggan, Jennie; Stonebraker, Michael.

SIGMOD 2014 - Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. Association for Computing Machinery, 2014. p. 409-420.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Incremental elasticity for array databases

AU - Duggan, Jennie

AU - Stonebraker, Michael

PY - 2014/1/1

Y1 - 2014/1/1

N2 - Relational databases benefit significantly from elasticity, whereby they execute on a set of changing hardware resources provisioned to match their storage and processing requirements. Such flexibility is especially attractive for scientific databases because their users often have a no-overwrite storage model, in which they delete data only when their available space is exhausted. This results in a database that is regularly growing and expanding its hardware proportionally. Also, scientific databases frequently store their data as multidimensional arrays optimized for spatial querying. This brings about several novel challenges in clustered, skewaware data placement on an elastic shared-nothing database. In this work, we design and implement elasticity for an array database. We address this challenge on two fronts: determining when to expand a database cluster and how to partition the data within it. In both steps we propose incremental approaches, affecting a minimum set of data and nodes, while maintaining high performance. We introduce an algorithm for gradually augmenting an array database's hardware using a closed-loop control system. After the cluster adds nodes, we optimize data placement for n-dimensional arrays. Many of our elastic partitioners incrementally reorganize an array, redistributing data only to new nodes. By combining these two tools, the scientific database efficiently and seamlessly manages its monotonically increasing hardware resources.

AB - Relational databases benefit significantly from elasticity, whereby they execute on a set of changing hardware resources provisioned to match their storage and processing requirements. Such flexibility is especially attractive for scientific databases because their users often have a no-overwrite storage model, in which they delete data only when their available space is exhausted. This results in a database that is regularly growing and expanding its hardware proportionally. Also, scientific databases frequently store their data as multidimensional arrays optimized for spatial querying. This brings about several novel challenges in clustered, skewaware data placement on an elastic shared-nothing database. In this work, we design and implement elasticity for an array database. We address this challenge on two fronts: determining when to expand a database cluster and how to partition the data within it. In both steps we propose incremental approaches, affecting a minimum set of data and nodes, while maintaining high performance. We introduce an algorithm for gradually augmenting an array database's hardware using a closed-loop control system. After the cluster adds nodes, we optimize data placement for n-dimensional arrays. Many of our elastic partitioners incrementally reorganize an array, redistributing data only to new nodes. By combining these two tools, the scientific database efficiently and seamlessly manages its monotonically increasing hardware resources.

UR - http://www.scopus.com/inward/record.url?scp=84904359199&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84904359199&partnerID=8YFLogxK

M3 - Conference contribution

SN - 9781450323765

SP - 409

EP - 420

BT - SIGMOD 2014 - Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data

PB - Association for Computing Machinery

ER -

Duggan J, Stonebraker M. Incremental elasticity for array databases. In SIGMOD 2014 - Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. Association for Computing Machinery. 2014. p. 409-420