AIDTN: Towards a Real-Time AI Optimized DTN System With NVMeoF

Se Young Yu*, Qingyang Zeng, Jim Chen, Yan Chen, Joe Mambretti

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Large-scale data transport for data-intensive sciences is a complex multidimensional challenge. The challenge includes optimizing the end-to-end Big Data movement performance in real-time, supporting direct remote data access using NVMe over Fabrics (NVMeoF) and deploying to existing research platforms. AIDTN is the first effort to provide a unique AI system designed to incorporate NVMe over Fabrics (NVMeoF) and optimize coordination among multiple components supporting large-scale, multi-domain Wide Area Network (WAN) data-intensive science. AIDTN's research objective is to integrate next-generation storage architecture using NVMeoF, specialized network design using high-performance network appliances, Data Transfer Nodes (DTNs), catalysts in driving data transport, and a unique AI system explicitly designed for high-performance data movement challenges. AIDTN is the first system that uses network and system features to predict the end-to-end performance of high-performance data movement and further extends the model with NVMe-specific features for NVMeoF remote data access. As a result, AIDTN improves data movement performance by up to 284% while minimizing packet loss compared to other heuristics approaches. It also has a prediction error rate as low as 0.16 compared to AI models with the only network (error rate = 0.29) or network and system features (error rate = 0.19).

Original languageEnglish (US)
Pages (from-to)1731-1742
Number of pages12
JournalIEEE Transactions on Parallel and Distributed Systems
Volume34
Issue number6
DOIs
StatePublished - Jun 1 2023

Keywords

  • Artificial intelligence
  • data transfer nodes
  • microservice
  • NVMe over fabrics

ASJC Scopus subject areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'AIDTN: Towards a Real-Time AI Optimized DTN System With NVMeoF'. Together they form a unique fingerprint.

Cite this