Parallel Implementation of Lossy Data Compression for Temporal Data Sets

Zheng Yuan; William Hendrix; Seung Woo Son; Christoph Federrath; Ankit Agrawal; Wei Keng Liao; Alok Choudhary

doi:10.1109/HiPC.2016.017

Parallel Implementation of Lossy Data Compression for Temporal Data Sets

Zheng Yuan, William Hendrix, Seung Woo Son, Christoph Federrath, Ankit Agrawal, Wei Keng Liao, Alok Choudhary

Electrical and Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Scopus citations

Abstract

Many scientific data sets contain temporal dimensions. These are the data storing information at the same spatial location but different time stamps. Some of the biggest temporal datasets are produced by parallel computing applications such as simulations of climate change and fluid dynamics. Temporal datasets can be very large and cost a huge amount of time to transfer among storage locations. Using data compression techniques, files can be transferred faster and save storage space. NUMARCK is a lossy data compression algorithm for temporal data sets that can learn emerging distributions of element-wise change ratios along the temporal dimension and encodes them into an index table to be concisely represented. This paper presents a parallel implementation of NUMARCK. Evaluated with six data sets obtained from climate and astrophysics simulations, parallel NUMARCK achieved scalable speedups of up to 8788 when running 12800 MPI processes on a parallel computer. We also compare the compression ratios against two lossy data compression algorithms, ISABELA and ZFP. The results show that NUMARCK achieved higher compression ratio than ISABELA and ZFP.

Original language	English (US)
Title of host publication	Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	62-71
Number of pages	10
ISBN (Electronic)	9781509054114
DOIs	https://doi.org/10.1109/HiPC.2016.017
State	Published - Feb 1 2017
Event	23rd IEEE International Conference on High Performance Computing, HiPC 2016 - Hyderabad, India Duration: Dec 19 2016 → Dec 22 2016

Publication series

Name	Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016

Other

Other	23rd IEEE International Conference on High Performance Computing, HiPC 2016
Country/Territory	India
City	Hyderabad
Period	12/19/16 → 12/22/16

Funding

This work is supported in part by the following grants: NSF awards CCF-1029166, IIS-1343639, CCF-1409601; DOE awards DE-SC0007456, DE-SC0014330; AFOSR award FA9550-12-1-0458; NIST award 70NANB14H012; DARPA award N66001-15-C-4036. C.F. acknowledges funding provided by the Australian Research Council's Discovery Projects (grants DP130102078 and DP150104329). C.F. further acknowledges supercomputing time provided by the Jülich Supercomputing Centre (grant hhd20), the Leibniz Rechenzentrum and the Gauss Centre for Supercomputing (grants pr32lo, pr48pi and GCS Large-scale project 10391), the Partnership for Advanced Computing in Europe (PRACE grant pr89mu), the Australian National Computational Infrastructure (grant ek9), and the Pawsey Supercomputing Centre with funding from the Australian Government and the Government of Western Australia. The simulation software FLASH was in part developed by the DOE-supported Flash Center for Computational Science at the University of Chicago.

Keywords

error-bound
lossy data compression
parallel data compression
temporal change ratio

ASJC Scopus subject areas

Hardware and Architecture

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/HiPC.2016.017

Cite this

Yuan, Z., Hendrix, W., Son, S. W., Federrath, C., Agrawal, A., Liao, W. K., & Choudhary, A. (2017). Parallel Implementation of Lossy Data Compression for Temporal Data Sets. In Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016 (pp. 62-71). Article 7839670 (Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HiPC.2016.017

Yuan, Zheng ; Hendrix, William ; Son, Seung Woo et al. / Parallel Implementation of Lossy Data Compression for Temporal Data Sets. Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 62-71 (Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016).

@inproceedings{5c148e9365cf4eb89d538073ba3d9024,

title = "Parallel Implementation of Lossy Data Compression for Temporal Data Sets",

abstract = "Many scientific data sets contain temporal dimensions. These are the data storing information at the same spatial location but different time stamps. Some of the biggest temporal datasets are produced by parallel computing applications such as simulations of climate change and fluid dynamics. Temporal datasets can be very large and cost a huge amount of time to transfer among storage locations. Using data compression techniques, files can be transferred faster and save storage space. NUMARCK is a lossy data compression algorithm for temporal data sets that can learn emerging distributions of element-wise change ratios along the temporal dimension and encodes them into an index table to be concisely represented. This paper presents a parallel implementation of NUMARCK. Evaluated with six data sets obtained from climate and astrophysics simulations, parallel NUMARCK achieved scalable speedups of up to 8788 when running 12800 MPI processes on a parallel computer. We also compare the compression ratios against two lossy data compression algorithms, ISABELA and ZFP. The results show that NUMARCK achieved higher compression ratio than ISABELA and ZFP.",

keywords = "error-bound, lossy data compression, parallel data compression, temporal change ratio",

author = "Zheng Yuan and William Hendrix and Son, {Seung Woo} and Christoph Federrath and Ankit Agrawal and Liao, {Wei Keng} and Alok Choudhary",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 23rd IEEE International Conference on High Performance Computing, HiPC 2016 ; Conference date: 19-12-2016 Through 22-12-2016",

year = "2017",

month = feb,

day = "1",

doi = "10.1109/HiPC.2016.017",

language = "English (US)",

series = "Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "62--71",

booktitle = "Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016",

address = "United States",

}

Yuan, Z, Hendrix, W, Son, SW, Federrath, C, Agrawal, A , Liao, WK & Choudhary, A 2017, Parallel Implementation of Lossy Data Compression for Temporal Data Sets. in Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016., 7839670, Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016, Institute of Electrical and Electronics Engineers Inc., pp. 62-71, 23rd IEEE International Conference on High Performance Computing, HiPC 2016, Hyderabad, India, 12/19/16. https://doi.org/10.1109/HiPC.2016.017

Parallel Implementation of Lossy Data Compression for Temporal Data Sets. / Yuan, Zheng; Hendrix, William; Son, Seung Woo et al.
Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016. Institute of Electrical and Electronics Engineers Inc., 2017. p. 62-71 7839670 (Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Parallel Implementation of Lossy Data Compression for Temporal Data Sets

AU - Yuan, Zheng

AU - Hendrix, William

AU - Son, Seung Woo

AU - Federrath, Christoph

AU - Agrawal, Ankit

AU - Liao, Wei Keng

AU - Choudhary, Alok

PY - 2017/2/1

Y1 - 2017/2/1

N2 - Many scientific data sets contain temporal dimensions. These are the data storing information at the same spatial location but different time stamps. Some of the biggest temporal datasets are produced by parallel computing applications such as simulations of climate change and fluid dynamics. Temporal datasets can be very large and cost a huge amount of time to transfer among storage locations. Using data compression techniques, files can be transferred faster and save storage space. NUMARCK is a lossy data compression algorithm for temporal data sets that can learn emerging distributions of element-wise change ratios along the temporal dimension and encodes them into an index table to be concisely represented. This paper presents a parallel implementation of NUMARCK. Evaluated with six data sets obtained from climate and astrophysics simulations, parallel NUMARCK achieved scalable speedups of up to 8788 when running 12800 MPI processes on a parallel computer. We also compare the compression ratios against two lossy data compression algorithms, ISABELA and ZFP. The results show that NUMARCK achieved higher compression ratio than ISABELA and ZFP.

AB - Many scientific data sets contain temporal dimensions. These are the data storing information at the same spatial location but different time stamps. Some of the biggest temporal datasets are produced by parallel computing applications such as simulations of climate change and fluid dynamics. Temporal datasets can be very large and cost a huge amount of time to transfer among storage locations. Using data compression techniques, files can be transferred faster and save storage space. NUMARCK is a lossy data compression algorithm for temporal data sets that can learn emerging distributions of element-wise change ratios along the temporal dimension and encodes them into an index table to be concisely represented. This paper presents a parallel implementation of NUMARCK. Evaluated with six data sets obtained from climate and astrophysics simulations, parallel NUMARCK achieved scalable speedups of up to 8788 when running 12800 MPI processes on a parallel computer. We also compare the compression ratios against two lossy data compression algorithms, ISABELA and ZFP. The results show that NUMARCK achieved higher compression ratio than ISABELA and ZFP.

KW - error-bound

KW - lossy data compression

KW - parallel data compression

KW - temporal change ratio

UR - http://www.scopus.com/inward/record.url?scp=85015155501&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85015155501&partnerID=8YFLogxK

U2 - 10.1109/HiPC.2016.017

DO - 10.1109/HiPC.2016.017

M3 - Conference contribution

AN - SCOPUS:85015155501

T3 - Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016

SP - 62

EP - 71

BT - Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 23rd IEEE International Conference on High Performance Computing, HiPC 2016

Y2 - 19 December 2016 through 22 December 2016

ER -

Yuan Z, Hendrix W, Son SW, Federrath C, Agrawal A , Liao WK et al. Parallel Implementation of Lossy Data Compression for Temporal Data Sets. In Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016. Institute of Electrical and Electronics Engineers Inc. 2017. p. 62-71. 7839670. (Proceedings - 23rd IEEE International Conference on High Performance Computing, HiPC 2016). doi: 10.1109/HiPC.2016.017

Parallel Implementation of Lossy Data Compression for Temporal Data Sets

Abstract

Publication series

Other

Funding

Keywords

ASJC Scopus subject areas

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this