Shore-MT: A scalable storage manager for the multicore era

Ryan Johnson*, Ippokratis Pandis, Nikos Hardavellas, Anastasia Ailamaki, Babak Falsafi

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

130 Citations (Scopus)

Abstract

Database storage managers have long been able to efficiently handle multiple concurrent requests. Until recently, however, a computer contained only a few single-core CPUs, and therefore only a few transactions could simultaneously access the storage manager's internal structures. This allowed storage managers to use non-scalable approaches without any penalty. With the arrival of multicore chips, however, this situation is rapidly changing. More and more threads can run in parallel, stressing the internal scalability of the storage manager. Systems optimized for high performance at a limited number of cores are not assured similarly high performance at a higher core count, because unanticipated scalability obstacles arise. We benchmark four popular open-source storage managers (Shore, BerkeleyDB, MySQL, and PostgreSQL) on a modern multicore machine, and find that they all suffer in terms of scalability. We briefly examine the bottlenecks in the various storage engines. We then present Shore-MT, a multithreaded and highly scalable version of Shore which we developed by identifying and successively removing internal bottlenecks. When compared to other DBMS, Shore-MT exhibits superior scalability and 2-4 times higher absolute throughput than its peers. We also show that designers should favor scalability to single-thread performance, and highlight important principles for writing scalable storage engines, illustrated with real examples from the development of Shore-MT.

Original languageEnglish (US)
Title of host publicationProceedings of the 12th International Conference on Extending Database Technology
Subtitle of host publicationAdvances in Database Technology, EDBT'09
Pages24-35
Number of pages12
DOIs
StatePublished - Sep 21 2009
Event12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09 - Saint Petersburg, Russian Federation
Duration: Mar 24 2009Mar 26 2009

Publication series

NameProceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09

Other

Other12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09
CountryRussian Federation
CitySaint Petersburg
Period3/24/093/26/09

Fingerprint

Scalability
Managers
Engines
Program processors
Throughput

ASJC Scopus subject areas

  • Computer Science Applications
  • Software

Cite this

Johnson, R., Pandis, I., Hardavellas, N., Ailamaki, A., & Falsafi, B. (2009). Shore-MT: A scalable storage manager for the multicore era. In Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09 (pp. 24-35). (Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09). https://doi.org/10.1145/1516360.1516365
Johnson, Ryan ; Pandis, Ippokratis ; Hardavellas, Nikos ; Ailamaki, Anastasia ; Falsafi, Babak. / Shore-MT : A scalable storage manager for the multicore era. Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09. 2009. pp. 24-35 (Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09).
@inproceedings{9ebf37ec9ad2495c8a15eb9f41e61ae3,
title = "Shore-MT: A scalable storage manager for the multicore era",
abstract = "Database storage managers have long been able to efficiently handle multiple concurrent requests. Until recently, however, a computer contained only a few single-core CPUs, and therefore only a few transactions could simultaneously access the storage manager's internal structures. This allowed storage managers to use non-scalable approaches without any penalty. With the arrival of multicore chips, however, this situation is rapidly changing. More and more threads can run in parallel, stressing the internal scalability of the storage manager. Systems optimized for high performance at a limited number of cores are not assured similarly high performance at a higher core count, because unanticipated scalability obstacles arise. We benchmark four popular open-source storage managers (Shore, BerkeleyDB, MySQL, and PostgreSQL) on a modern multicore machine, and find that they all suffer in terms of scalability. We briefly examine the bottlenecks in the various storage engines. We then present Shore-MT, a multithreaded and highly scalable version of Shore which we developed by identifying and successively removing internal bottlenecks. When compared to other DBMS, Shore-MT exhibits superior scalability and 2-4 times higher absolute throughput than its peers. We also show that designers should favor scalability to single-thread performance, and highlight important principles for writing scalable storage engines, illustrated with real examples from the development of Shore-MT.",
author = "Ryan Johnson and Ippokratis Pandis and Nikos Hardavellas and Anastasia Ailamaki and Babak Falsafi",
year = "2009",
month = "9",
day = "21",
doi = "10.1145/1516360.1516365",
language = "English (US)",
isbn = "9781605584225",
series = "Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09",
pages = "24--35",
booktitle = "Proceedings of the 12th International Conference on Extending Database Technology",

}

Johnson, R, Pandis, I, Hardavellas, N, Ailamaki, A & Falsafi, B 2009, Shore-MT: A scalable storage manager for the multicore era. in Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09. Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09, pp. 24-35, 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09, Saint Petersburg, Russian Federation, 3/24/09. https://doi.org/10.1145/1516360.1516365

Shore-MT : A scalable storage manager for the multicore era. / Johnson, Ryan; Pandis, Ippokratis; Hardavellas, Nikos; Ailamaki, Anastasia; Falsafi, Babak.

Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09. 2009. p. 24-35 (Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Shore-MT

T2 - A scalable storage manager for the multicore era

AU - Johnson, Ryan

AU - Pandis, Ippokratis

AU - Hardavellas, Nikos

AU - Ailamaki, Anastasia

AU - Falsafi, Babak

PY - 2009/9/21

Y1 - 2009/9/21

N2 - Database storage managers have long been able to efficiently handle multiple concurrent requests. Until recently, however, a computer contained only a few single-core CPUs, and therefore only a few transactions could simultaneously access the storage manager's internal structures. This allowed storage managers to use non-scalable approaches without any penalty. With the arrival of multicore chips, however, this situation is rapidly changing. More and more threads can run in parallel, stressing the internal scalability of the storage manager. Systems optimized for high performance at a limited number of cores are not assured similarly high performance at a higher core count, because unanticipated scalability obstacles arise. We benchmark four popular open-source storage managers (Shore, BerkeleyDB, MySQL, and PostgreSQL) on a modern multicore machine, and find that they all suffer in terms of scalability. We briefly examine the bottlenecks in the various storage engines. We then present Shore-MT, a multithreaded and highly scalable version of Shore which we developed by identifying and successively removing internal bottlenecks. When compared to other DBMS, Shore-MT exhibits superior scalability and 2-4 times higher absolute throughput than its peers. We also show that designers should favor scalability to single-thread performance, and highlight important principles for writing scalable storage engines, illustrated with real examples from the development of Shore-MT.

AB - Database storage managers have long been able to efficiently handle multiple concurrent requests. Until recently, however, a computer contained only a few single-core CPUs, and therefore only a few transactions could simultaneously access the storage manager's internal structures. This allowed storage managers to use non-scalable approaches without any penalty. With the arrival of multicore chips, however, this situation is rapidly changing. More and more threads can run in parallel, stressing the internal scalability of the storage manager. Systems optimized for high performance at a limited number of cores are not assured similarly high performance at a higher core count, because unanticipated scalability obstacles arise. We benchmark four popular open-source storage managers (Shore, BerkeleyDB, MySQL, and PostgreSQL) on a modern multicore machine, and find that they all suffer in terms of scalability. We briefly examine the bottlenecks in the various storage engines. We then present Shore-MT, a multithreaded and highly scalable version of Shore which we developed by identifying and successively removing internal bottlenecks. When compared to other DBMS, Shore-MT exhibits superior scalability and 2-4 times higher absolute throughput than its peers. We also show that designers should favor scalability to single-thread performance, and highlight important principles for writing scalable storage engines, illustrated with real examples from the development of Shore-MT.

UR - http://www.scopus.com/inward/record.url?scp=70349141254&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70349141254&partnerID=8YFLogxK

U2 - 10.1145/1516360.1516365

DO - 10.1145/1516360.1516365

M3 - Conference contribution

AN - SCOPUS:70349141254

SN - 9781605584225

T3 - Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09

SP - 24

EP - 35

BT - Proceedings of the 12th International Conference on Extending Database Technology

ER -

Johnson R, Pandis I, Hardavellas N, Ailamaki A, Falsafi B. Shore-MT: A scalable storage manager for the multicore era. In Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09. 2009. p. 24-35. (Proceedings of the 12th International Conference on Extending Database Technology: Advances in Database Technology, EDBT'09). https://doi.org/10.1145/1516360.1516365