Distributed MARL for Scheduling in Conflict Graphs

Yiming Zhang; Dongning Guo

doi:10.1109/Allerton58177.2023.10313455

Distributed MARL for Scheduling in Conflict Graphs

Yiming Zhang^*, Dongning Guo

^*Corresponding author for this work

Electrical and Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Scopus citations

Abstract

This paper addresses a link scheduling problem in networks represented by conflict graphs using a distributed learning approach. Each agent in a network controls a single link and has access only to its own state and the states of links in its neighborhood. The goal is to minimize the average packet delay in the multi-agent network. The problem is formulated as a decentralized partially observable Markov decision process (Dec-POMDP). The proposed solution adopts a centralized training and distributed execution paradigm and leverages an on-policy reinforcement learning algorithm. Specifically, the paper employs the multi-agent proximal policy optimization (MAPPO) algorithm with judiciously designed recurrent structures in the neural network. The proposed solution is shown to outperform some widely used schedulers in terms of throughput and delays through simulations.

Original language	English (US)
Title of host publication	2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9798350328141
DOIs	https://doi.org/10.1109/Allerton58177.2023.10313455
State	Published - 2023
Event	59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023 - Monticello, United States Duration: Sep 26 2023 → Sep 29 2023

Publication series

Name	2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023

Conference

Conference	59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023
Country/Territory	United States
City	Monticello
Period	9/26/23 → 9/29/23

Funding

The work was supported in part by the National Science Foundation under grant No. 2003098, a gift from Intel Corporation, the SpectrumX Center (NSF grant No. 2132700), and also the Institute for Data, Econometrics, Algorithms and Learning (NSF grant No. 2216970). This work was supported in part by NSF under Contract Nos. CNS-1719384 and IIS-1636772.

Keywords

Decentralized partially observable Markov decision process (Dec-POMDP)
dynamic traffic
multi-agent reinforcement learning (MARL)
recurrent neural networks
wireless networks

ASJC Scopus subject areas

Artificial Intelligence
Computational Theory and Mathematics
Computer Networks and Communications
Computer Science Applications
Computational Mathematics
Control and Optimization

Access to Document

10.1109/Allerton58177.2023.10313455

Cite this

Zhang, Y., & Guo, D. (2023). Distributed MARL for Scheduling in Conflict Graphs. In 2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023 (2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/Allerton58177.2023.10313455

@inproceedings{ff876d604fd64b5da9c6f4e82610319b,

title = "Distributed MARL for Scheduling in Conflict Graphs",

abstract = "This paper addresses a link scheduling problem in networks represented by conflict graphs using a distributed learning approach. Each agent in a network controls a single link and has access only to its own state and the states of links in its neighborhood. The goal is to minimize the average packet delay in the multi-agent network. The problem is formulated as a decentralized partially observable Markov decision process (Dec-POMDP). The proposed solution adopts a centralized training and distributed execution paradigm and leverages an on-policy reinforcement learning algorithm. Specifically, the paper employs the multi-agent proximal policy optimization (MAPPO) algorithm with judiciously designed recurrent structures in the neural network. The proposed solution is shown to outperform some widely used schedulers in terms of throughput and delays through simulations.",

keywords = "Decentralized partially observable Markov decision process (Dec-POMDP), dynamic traffic, multi-agent reinforcement learning (MARL), recurrent neural networks, wireless networks",

author = "Yiming Zhang and Dongning Guo",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023 ; Conference date: 26-09-2023 Through 29-09-2023",

year = "2023",

doi = "10.1109/Allerton58177.2023.10313455",

language = "English (US)",

series = "2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023",

address = "United States",

}

Zhang, Y & Guo, D 2023, Distributed MARL for Scheduling in Conflict Graphs. in 2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023. 2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023, Institute of Electrical and Electronics Engineers Inc., 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023, Monticello, United States, 9/26/23. https://doi.org/10.1109/Allerton58177.2023.10313455

Distributed MARL for Scheduling in Conflict Graphs. / Zhang, Yiming; Guo, Dongning.
2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Distributed MARL for Scheduling in Conflict Graphs

AU - Zhang, Yiming

AU - Guo, Dongning

PY - 2023

Y1 - 2023

N2 - This paper addresses a link scheduling problem in networks represented by conflict graphs using a distributed learning approach. Each agent in a network controls a single link and has access only to its own state and the states of links in its neighborhood. The goal is to minimize the average packet delay in the multi-agent network. The problem is formulated as a decentralized partially observable Markov decision process (Dec-POMDP). The proposed solution adopts a centralized training and distributed execution paradigm and leverages an on-policy reinforcement learning algorithm. Specifically, the paper employs the multi-agent proximal policy optimization (MAPPO) algorithm with judiciously designed recurrent structures in the neural network. The proposed solution is shown to outperform some widely used schedulers in terms of throughput and delays through simulations.

AB - This paper addresses a link scheduling problem in networks represented by conflict graphs using a distributed learning approach. Each agent in a network controls a single link and has access only to its own state and the states of links in its neighborhood. The goal is to minimize the average packet delay in the multi-agent network. The problem is formulated as a decentralized partially observable Markov decision process (Dec-POMDP). The proposed solution adopts a centralized training and distributed execution paradigm and leverages an on-policy reinforcement learning algorithm. Specifically, the paper employs the multi-agent proximal policy optimization (MAPPO) algorithm with judiciously designed recurrent structures in the neural network. The proposed solution is shown to outperform some widely used schedulers in terms of throughput and delays through simulations.

KW - Decentralized partially observable Markov decision process (Dec-POMDP)

KW - dynamic traffic

KW - multi-agent reinforcement learning (MARL)

KW - recurrent neural networks

KW - wireless networks

UR - http://www.scopus.com/inward/record.url?scp=85179505732&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85179505732&partnerID=8YFLogxK

U2 - 10.1109/Allerton58177.2023.10313455

DO - 10.1109/Allerton58177.2023.10313455

M3 - Conference contribution

AN - SCOPUS:85179505732

T3 - 2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023

BT - 2023 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 59th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2023

Y2 - 26 September 2023 through 29 September 2023

ER -

Distributed MARL for Scheduling in Conflict Graphs

Abstract

Publication series

Conference

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this